Due to the problems of targets submerged to the background, which could easily produce ghosts, and hard complete extraction of dim targets in the surveillance video, we propose the moving target extraction and fast video reconstruction algorithm in accord with visual principle. The sample selection strategy of VIBE algorithm is improved to alleviate the errors of pixel classification. The infrared imaging features are fused to suppress the artifact. A regional growth mechanism is established to extract and store moving targets and pure background regions, and according to the characteristics of video surveillance, it is the first to establish the mapping mechanism of target, background and video to propose the fast video reconstruction algorithm. The experiment shows that the algorithm can extract the moving target completely, establish the pure background in a variety of complex conditions, and greatly reduce the storage room of the surveillance video.