Tea is the main commercial crop grown in China, and excessive application of chemical fertilizers in tea plantations is common. However, the potential to reduce chemical fertilizer use in tea plantations is unclear. In this study, Zhejiang Province was selected as the research object to systematically analyze the potential for tea plantation chemical-fertilizer reduction at different spatial scales. The geographic information system-based analytic hierarchy process method and Soil and Water Assessment Tool model were used to determine the chemical fertilizer reduction potential at the province scale and watershed scale, respectively. At the field scale, two consecutive years of field experiments were conducted on a tea plantation. Province-level analysis showed that 51.7% of the area had an average total fertilization intensity greater than 350 kg/hm2 and a high reduction potential. Watershed analysis revealed that chemical fertilizer reduction had better potential in reducing total nitrogen and total phosphorus inputs to runoff in the short term, whereas 50% organic fertilizer substitution was the best strategy to achieve long-term effects. The field experiments further proved that organic fertilizer substitution balanced tea growth and environmental protection. This study provides a useful method to investigate strategies to reduce chemical fertilizer use in tea-growing areas.