The optical system is one of the core components for star sensors, whose imaging quality directly influences the performance of star sensors for star detection, thereby determining the attitude control accuracy of spacecrafts. Here, we report a new type of optical system with a catadioptric structure and a large entrance pupil diameter for a 15-magnitude star sensor. It consists of an improved Cassegrain system (R-C system), an aperture correction spherical lens group and a field of view correction spherical lens group. By embedding the secondary mirror of the R-C system into the output surface of the negative spherical lens of the aperture correction spherical lens group, the blocking of incident light is eliminated from the secondary mirror holder. After the structure optimization, the catadioptric optical system (COS) had a spectral range of 450 nm950 nm, an entrance pupil diameter of 250 mm, a half-diagonal field of view of 1.4 and a focal length of 390 mm. By using theoretical calculations and experimental measurements, it was verified that the COS, with the ability to correct astigmatism, lateral color and distortion, can fulfill the detection of 15-magnitude dark stars. © 2020 by the authors. Licensee MDPI, Basel, Switzerland.