Basing on the momentum mismatching together with the crystal rotatory property, we investigate the diffraction performance of an acousto-optic tunable filter (AOTF). The relationship between diffraction efficiency, momentum mismatching, incident optical wave vector, and ultrasound wave vector is analyzed. The correlation between the frequency tuning relation and the incident angle of light is demonstrated, which is usually ignored in the AOTF design. The diffracted wavelength can be decided by a particular acoustic frequency only when the incident angle of light is fixed. Theoretical and experimental analysis indicates that we can acquire a narrowband spectrum and big angular aperture with a proper incident angle. © 2018 Optical Society of America.