According to the principle of thermal imaging, moving targets can be better located in infrared images, but their boundary is blurred, and the details of objects cannot be displayed. The details of objects in natural images can be better shown, but for the condition of shelter, shadow and etc., miss-tracking and false-tracking may easily occur. Thus, we construct a framework for moving target extraction and tracking in infrared and natural images. For infrared images: according to the rough fuzzy set theory, we propose the rough entropy model based on the traditional frame difference method. The model is fused with the infrared imaging characteristics to locate moving target regions. For natural images: a time-space fusion LBP model is proposed for target coding. The model is integrated into the GMM model to obtain moving target information. The moving regions in infrared images and natural images are fused to form a priori model, and the C-V model is improved to extract targets accurately. © 2019