At present, there are many excellent algorithms in the field of visual object tracking. The correlation filter algorithms are more suitable for infrared object tracking, for the tracking performance is the best. Especially, the long-term correlation tracking has received much attention, owing to its ability to handle the problems of universal tracking (e.g., slight deformation, small-displacement motion, partial occlusion and out of view). However, there are three imperfections such that it fails to solve the problem of rapid motion, it cannot cope with the problem of boundary effect perfectly, and it has poor tracking effect in the case of severe occlusion and severe deformation. Aiming at the problem of rapid motion, the FlowNet 2.0 is introduced to accomplish optical flow estimation, offering motion information and predicting trajectory change process. Aiming to address the concern of boundary effect perfectly, the adjustable Gaussian window is effective to separate the object and the background, improving classifier discrimination. Aiming at the issue of poor tracking effect in the case of severe occlusion and severe deformation, the SURF feature-based matching method is effective to accurately track object and improve the infrared object tracking performance. In addition, the ratio between average peak-tocorrelation energy (APCE) and its historical average, as a further complement of maximum response, is introduced to achieve the online updating mechanism, jointly determining whether the tracker needs to be updated, the SURF matching needs to be carried out or the tracker needs to be initialized by re-detector. Our algorithm is validated on synthetic infrared aerial object image sequences, real infrared thermal aerial object image sequences and a public database named AMCOM FLIR respectively. The extensive experimental testify that our improved approach achieves an optimal effect for aerial infrared object tracking in terms of precision plot, success plot and speed.