In this paper, the double-layer harmonic diffractive element (HDE) structure is investigated and the optimization procedure is based on the equation of diffraction efficiency of the double-layer diffractive optical element. The diffraction efficiency of the system in the designed middle and far infrared wavebands is larger than 99%, which improves the image contrast and the imagequality significantly. A new dual-band infrared double-layer HDE telescope is designed, which can work in the middle and far infrared wavebands. It is shown that the system approximately attains diffraction limit and is easy to processed.