Extensive mineral exploitation activities in history have aggravated potential toxic elements (PTEs) contamination in agricultural soils in China. Comprehensive ecological risk assessment is of great significance to orientate the restoration of contaminated soils, especially for those with high background values and multiple sources. The study area is located in the major rice producing area of China. Historically, there was a silver mine and a lead-zinc mine in the area, which were successively closed during the investigation. The intensive mining activities caused serious PTEs pollution in the agricultural soils around the mining area. In this study, five PTEs (As, Cd, Cr, Hg and Pb) selected to assessed the potential of geoaccumulation index in assessing agricultural soil potential risk assessment by identifying ecological risk sources. 315 of soil samples collected in 2009, 2014, 2018 were comprehensively analyzed by single pollution index evaluation (single factor index, geoaccumulation index), comprehensive evaluation (Nemerow index, potential ecological risk index) and trend analysis. Single factor index analysis showed that geoaccumulation index considered the impact of natural diagenesis of background values and human activities on the environment, ensuring high evaluation accuracy comparing to other methods used in typical complex agricultural soils. The modified potential ecological risk index revealed that the high background area did not represent high risk area, which was consistent with the implementation effect of governance measures. This study can provide important insights for policymakers and environmental engineers to quantitatively recognize the soil pollution and the effectiveness of governance based on applicable and reasonable evaluation methods.