Since space cameras need to withstand the harsh mechanical and thermal conditions in the space environment for a long time, it is necessary to calibrate them in orbit. However, existing calibration methods have various disadvantages, making them impossible to use in orbit. To address this problem, we present an on-orbit calibration of space camera lens distortion with the vanishing points obtained from a single image of the solar panel. First, we propose a parallel-line-extraction method based on collinear constraints to obtain the parallel lines. Then, we train the optimal vanishing point using the common point constraint method. Using the optimal vanishing point, we establish the optimization function of lens distortion based on vanishing point consistency. Finally, we present an improved genetic algorithm to solve the optimization function. Simulations and experiments show that the proposed method is flexible and robust.