The polarization characteristics of an astronomical telescope is an important factor that affects polarimetry accuracy. Polarization modeling is an essential means to achieve high precision and efficient polarization measurement of the telescope, especially for the alt-azimuth mount telescope. At present, the polarization model for the telescope (i.e., the physical parametric model) is mainly constructed using the polarization parameters of each optical element. In this paper, an artificial neural network (ANN) is used to model the polarization characteristics of the telescope. The ANN model between the physical parametric model residual and the pointing direction of the telescope is obtained, which reduces the model deviation caused by the incompleteness of the physical parametric model. Compared with the physical parametric model, the model fitting and predictive accuracy of the New Vacuum Solar Telescope (NVST) is improved after adopting the ANN model. After using the ANN model, the polarization cross-talk from I to Q, U, and V can be reduced from 0.011 to 0.007, and the crosstalk among Q, U, and V can be reduced from 0.047 to 0.020, which effectively improves the polarization measurement accuracy of the telescope.