Localization technology plays a key role in autonomous driving. Stereo visual odometry is a meaningful visual localization method to estimate the pose of autonomous vehicles. VINS-Fusion provides a state-of-the-art stereo visual odometry with Kanade-Lucas-Tomasi (KLT) tracker to achieve fast feature tracking. However, KLT tracker is prone to fall into local minima in urban environments due to illumination changes and large displacements, leading to catastrophic cumulative drift over time. Aiming to solve this problem, we present a light and adaptive feature tracking technique for VINS-Fusion to get a reliable set of measurements for pose estimation. First, a disparity constraint is incorporated into left-right check to refine the measurements. Next, we propose a light bi-circular check to further remove outliers, which has high efficiency with the ingenious design. Additionally, an adaptive strategy for feature selection is proposed to dynamically balance the quantity and quality of the measurements. Experiments demonstrate that our method outperforms VINS-Fusion by producing more accurate pose estimation with 20% speedup on the KITTI odometry benchmark.