Semantic segmentation plays a pivotal role in interpreting high-resolution remote sensing images (RSIs), where contextual information is essential for achieving accurate segmentation. Despite the common practice of partitioning large RSIs into smaller patches for deep model input, existing methods often rely on adaptations from natural image semantic segmentation techniques, limiting their contextual scope to individual images. To address this limitation and harness a broader range of contextual information from original large-scale RSIs, this study introduces a global feature fusion network (GFFNet). GFFNet employs a novel approach by incorporating a group transformer structure alternated with group convolution, forming a lightweight global context learning branch. This design facilitates the extraction of global contextual features from the large-scale RSIs. In addition, we propose a cross feature fusion module that seamlessly integrates local features obtained from the convolutional network with the global contextual features. GFFNet serves as a versatile plugin for existing RSI semantic segmentation models, particularly beneficial when the target dataset involves cropping. This integration enhances the model's performance, especially in terms of segmenting large-scale objects. Experimental results on the ISPRS and GID-15 datasets validate the effectiveness of GFFNet in improving segmentation capabilities for large scale objects in RSIs.