Infrared (IR) images can distinguish salient targets from their backgrounds based on the radiation difference in all-weather conditions. By contrast, visible (VIS) images can contain high-resolution texture and details information, which is more suitable for human observation. Therefore, it is quite necessary to combine both imaging advantages of these two kinds of images. Compared with the existing methods, we believe that scale decomposition based methods are the most active and efficient image fusion methods, which also have the best fusion effects. Inspired by the present scale decomposition methods, we propose a new feature decomposition method. Firstly, we propose an improved guided filter called edge-preserving guided filter (EPGF), which adopts the image gradient map for further improving the filtering effect. Subsequently, by using the EPGF, we decompose the IR and VIS images into three kinds of layers, including salient feature layers, luminance layers and detail layers. At the same time, we combine all the layers together to get an initial fusion result. Finally, we optimize the initial fusion image according to a new image fusion optimization model and ADMM, and the final fusion result will be obtained after several iterations. Compared with other scale decomposition methods, our proposed feature decomposition based method takes the IR salient targets, IR and VIS background illumination, as well as VIS details into consideration which is more in line with human visual observation, besides the computational efficiency is also superior. Experimental results indicate that this method has better subjective and objective evaluation results compared with other state-of-the-art methods. © 2021 Elsevier B.V.