Change detection (CD) is a crucial task in remote sensing (RS) image analysis. In recent years, the development of deep learning has led to significant progress in this field. However, current deep learning-based methods struggle to achieve accurate change detection in complex scenes, often resulting in false detections and loss of details of change objects. In this paper, we propose a novel granular-level and layer-level multi-scale fusion network (GLMF-Net) to overcome these problems. The GLMF-Net consists of two key modules: the granular-level multi-scale fusion (GMF) module and the layer-level multi-scale fusion (LMF) module. The GMF module locates potential change objects by capturing granular-level change features, while the LMF module excavates the details of change objects in shallow features. To achieve inter-layer feature fusion, we also develop a group-wise guidance operation in the LMF module. Extensive experimental results demonstrate that our GLMF-Net significantly improves the accuracy of change detection in complex scenes, and achieves the state-of-the-art performance on the widely used CDD and LEVIR-CD datasets in terms of five standard metrics. © 2023 IEEE.