Precise brain tumor segmentation can improve patient prognosis. However, due to the complicated structure of the human brain, brain tumor segmentation is a challenging task. To improve the brain tumor segmentation performance, a group cross-channel attention residual UNet (GCAUNet) that can make full use of the low-level fine details of tumor regions is proposed. First, iterative background removal and image normalization are applied to remove the disturbances of the background and illumination variations, respectively. Then, GCAUNet is constructed for brain tumor segmentation. To recover the fine details of brain tumors, a parallel network path, namely, the detail recovering (DR) path, is introduced to extract detail feature groups from multiscale low-level feature maps. In addition, to emphasize the significant feature groups and channels, a coarse-to-fine cross channel attention module, namely, the group cross-channel attention (GCA) module, is proposed. Furthermore, a multiscale input (MI) path is introduced into GCAUNet to acquire multiscale context information. The experimental results show that the average dices of GCAUNet for the whole tumor (WT), tumor core (TC) and enhancing tumor (ET) on BraTS 2017 and BraTS 2018 reach 87.2%/79.6%/78.1% and 90.9%/84.5%/81.3%, respectively. Compared with the backbone, the dices of WT, TC and ET on BraTS 2017 and 2018 are improved by 4.8%/7.1%/6.2% and 4.1%/3.5%/3.1%, respectively, which indicates that GCAUNet can significantly improve the brain tumor segmentation performance.