A segmentation method to separate the moving objects from the background in a generic video sequence is proposed. Firstly, the global motion estimation and compensation and the scene-change detection are achieved. Secondly, a robust discrimination between background and foreground by means of a higher-order statistical significance test is performed on a group of inter-frame differences, and a binary segmentation map is produced by means of a connected component labeling and a cascade of morphological filters. Thirdly, the symmetrical difference method is used to eliminate the effect of uncovered and covered background. At last,the mask matching and updating enable the algorithm to handle both objects that change quickly and objects that temporarily stop moving.