Road damage detection (RDD) is critical to society's safety and the efficient allocation of resources. Most road damage detection methods which directly adopt various object detection models face some significant challenges due to the characteristics of the RDD task. First, the damaged objects in the road images are highly diverse in scales and difficult to differentiate, making it more challenging than other tasks. Second, existing methods neglect the relationship between the feature distribution and model structure, which makes it difficult for optimization. To address these challenges, this study proposes an efficient dense attention fusion network with channel correlation loss for road damage detection. First, the K-Means++ algorithm is applied for data preprocessing to optimize the initial cluster centers and improve the model detection accuracy. Second, a dense attention fusion module is proposed to learn spatial-spectral attention to enhance multi-scale fusion features and improve the ability of the model to detect damage areas at different scales. Third, the channel correlation loss is adopted in the class prediction process to maintain the separability of intra and inter-class. The experimental results on the collected RDDA dataset and RDD2022 dataset show that the proposed method achieves state-of-the-art performance.