In China, energy use from passenger vehicles is increasing rapidly due to urbanization and income growth, but the contributions of transport service structure shift and energy efficiency improvement, especially future energy-saving directions are controversial and unclear because of the differences in the choices of traffic modes, which may greatly affect the analysis results and their interpretations. This study investigates the growth sources of energy use in the passenger transport sector during 1979–2017 and future energy-saving potentials by 2030 by re-examining statistical data. Results show that the transport service increase and transport structure shift led to the increase in energy use by 665.25% and 337.22%, but approximately 14% of the increased energy use was offset by energy efficiency improvements, and its effect is limited in public transportation sector, compared with private passenger vehicle sector. In public passenger transport sector, adjusting the service or volume structure may be a desirable option while in private passenger vehicle sector the policy focus is more on improving fuel efficiency and reducing average annual kilometers traveled per capita. Overall, there is still at least 28.3% of energy reduction potential by energy efficiency improvement before 2030 compared to 2017 baseline level. Potential evaluation showed that the current potential of transport service structure adjustment based on industrial planning is limited, which highlights the necessity of reform, involving promoting the sharing transportation, railway, subway and buses.