In conventional nuclear propulsion systems, reactor with gas-cooled core is bulky and requires extra space to be placed. A new type of nuclear heat exchanger with multi-annular structure is directly put into the combustion chamber to heat compressed air. Nuclear fuel is evenly embedded into the annular wall, which guarantees a steady energy supply to meet the propulsion needs. Because of the strong heat transfer capacity, the reactor is smaller, more compact and flexible. In this paper, a numerical model of propulsion and heat transfer is established for the nuclear jet engine to verify the feasibility of the novel nuclear heat exchanger. To find the optimal structure for cruise and after-burning flight, flow and heat transfer characteristics of the new reactor system are studied by adjusting the annulus numbers in the finite space. The sensitivity of the external control variables to the improvement of propulsion performance is investigated under the new structure. Finally, reactor performance is analyzed by physical-thermal coupling calculation in core. The analysis results show that the multi-annular nuclear heat exchanger has the optimal structure at the intersection point of characteristic parameter normalization. The multi-annular structure with n = 64 has more uniform power distribution, which is selected as the optimal structure for the new system. It can meet the heat transfer and thrust requirements from cruise to after-burning flight. The new system reduces the additional reactor configuration and saves the afterburner space, which is of great importance to the miniaturization and performance improvement of the nuclear propulsion system. (C) 2018 Elsevier Ltd. All rights reserved.