Constrained layer damping treatment is widely used to suppress the vibration and noise of thin-walled structures. However, full coverage of constrained damping layer will increase unnecessary additional mass, resulting in material waste and cannot effectively improve the damping performance of the composite structure. In this article, a topology optimization approach is proposed to realize the optimal distribution of constrained damping layer. The design objective is to maximize modal loss factors solved by the modal strain energy method under the constraint of volume. Taking the relative density of the finite element of the constrained damping layer as design variable, the solid isotropic material with penalization method is used to realize the optimal topological distribution of the damping material on the surface of the metal substrate. Then the moving asymptote method is adopted as an optimizer to search the optimal layout of the constrained damping layer. Based on a modified modal superposition method, the sensitivities of the objective function with respect to the design variables are obtained. Numerical examples and experiments are presented for illustrating the validity and efficiency of this approach. The results show that the objective function converges to the optimal value smoothly, and the optimized modal loss factors have been significantly improved. The layouts of the constrained damping layer after optimization are clear and reasonable, and its distributions are affected by both the damping layer and the constraining layer. Each part of the constrained damping layer after optimizing can greatly improve the damping performance of the structure.