Complex optical systems are composed of multiple subsystem modules, and high-precision registration between modules is the key to ensuring the overall performance indicators of optical systems. The registration between submodules in the engineering optical system is divided into two aspects: spatial geometric coordination and optical feature registration. The relationship between spatial geometric alignment and optical feature registration is introduced, and the spatial geometric registration between subsystems is the linear relationship. The subsystem model is established using data matrix, and the subsystem data matrix and physical entities are mapped to each other to form the digital twin. The registration between subsystems with each other applies matrix changes to establish the mathematical model, and each subsystem forms a large complex engineering optical system through registration. The error affecting the registration performance of the subsystem, the relationship between the registration error and the performance of the entire optical system, and the error data transfer between the physical entity and the digital matrix model were systematically analyzed, as well as the mathematical model and the entity registration scheme were optimized and corrected. © 2023 SPIE.