Under the background of large-scale ecological restoration, China’s southwestern karst region has become a hotspot of global vegetation cover “greening” in the past 20 years. However, because of geological constraints, it is difficult to restore the forest landscapes in some areas. It is urgent to understand the impacts of human disturbances during the historical period on the difficult-to-forestation rocky-desertification areas of the karst region in order to guide future afforestation. In this study, we quantified the changes of specific sediment yield in typical karst depressions over the past 500 years by using ¹³⁷Cs, ²¹⁰Pb, and ¹⁴C dating methods in karst depressions, and identified the main human disturbances related to historical erosion and sedimentation by combining with historical data. The results showed that the erosion and sedimentation of the three depressions in 1921–1963 were significantly higher than that in 1963–2021, and the sedimentation rate (0.64–1.33 cm a⁻¹) and the specific sediment yield (2.51–13.11 t ha⁻¹ a⁻¹) during Ming and Qing dynasties (1504–1812) were higher than the sedimentation rate (0.26–0.95 cm a⁻¹) and specific sediment yield (0.95–6.99 t ha⁻¹ a⁻¹) in the recent century (1921–2021). Reconstruction data and literature from the Ming and Qing dynasties show an empirical link between changes in population, arable land, food, forest area, and deforestation events during the same period. It was found that the population and arable land in Guangxi increased more than three-fold and the forest area decreased significantly in the 17th century after the migration of the Yao ethnic group and the introduction of maize to the region, which may be the main reason for intensifying the erosion of depressions. This study is of great significance to understanding the evolution history of rocky desertification in this region and to answer the potential of afforestation.