Purpose Accurate localization of radioactive materials is critical to nuclear safety and nuclear security. A coded aperture imaging system provides a visualization solution. However, the correlation method has poor reconstruction performance for sources with low counts and for extended sources. Methods In this study, a Monte Carlo optimization-based MLEM algorithm (MC-MLEM) is proposed. The system matrix was obtained by accurate Monte Carlo simulation, so the physical effects such as mask penetration that affect the imaging process were taken into account in the MLEM algorithm. In the simulation process, the normalization of the system matrix was realized by controlling the source at different position of the source plane to have the same activity and emission angle. Results The experimental results showed that compared with the correlation method, the MC-MLEM algorithm could improve the signal-to-noise ratio and angular resolution and locate the source position quickly and accurately under low count conditions. Furthermore, the MC-MLEM algorithm could reconstruct the shape of the extended source and the expected activity ratio of cold-hot sources with large activity differences. Conclusion The MC-MLEM algorithm improved the imaging results and enhanced the reconstruction performance.