As a high-precision position actuator, the voice coil motor (VCM) is widely used in technology of segmented mirror and mirror shape correction in space telescope. In response to the adverse effects of internal and external disturbances on the precision positioning of the VCM, an improved particle swarm optimization (PSO) based active disturbance rejection control (ADRC) VCM position control model is proposed in this paper. Our model is able to present a reasonable solution to the problem that ADRC has numerous parameters which are knotty to configure. In order to prevent precocious convergence or falling into local optimal, a variable weight factor search method is proposed. MATLAB and Simulink are employed for co-simulation and control experiment are designed. The simulation results show that compared with traditional PID control, the control model designed in this paper has better dynamic performance, higher tracking accuracy and better disturbance rejection performance, and it is suitable for high-precision position control under complex environment with a bright engineering application prospect.