Scratches on the optical surface can directly affect the reliability of the optical system. Machine vision-based methods have been widely applied in various industrial surface defect inspection scenarios. Since weak scratches imaging in the dark field has an ambiguous edge and low contrast, which brings difficulty in automatic defect detection. Recently, many existing visual inspection methods based on deep learning cannot effectively inspect weak scratches due to the lack of attention-aware features. To address the problems arising from industry-specific characteristics, this paper proposes “Attention Fusion Network;”, a convolutional neural network using attention mechanism built by hard and soft attention modules to generate attention-aware features. The hard attention module is implemented by integrating the brightness adjustment operation in the network, and the soft attention module is composed of scale attention and channel attention. The proposed model is trained on a real-world industrial scratch dataset and compared with other defect inspection methods. The proposed method can achieve the best performance to detect the weak scratch inspection of optical components compared to the traditional scratch detection methods and other deep learning-based methods.