Semantic segmentation is in-demand in High Resolution Remote Sensing (HRRS) image processing. Unlike natural images, HRRS images usually provide channels such as Near Infrared (NIR) in addition to RGB channels. However, in order to make use of the pre-trained model, the current semantic segmentation methods in remote sensing field usually only use the RGB channel and discard the information of other channels. In this paper, to make full use of the HRRS image information, a dual-stream fusion network is proposed to fuse the information of different channel combinations through a Feature Pyramid Network (FPN), then a Stage Pyramid Pooling (SPP) module is used to integrate the features of different scales and produce the final segmentation results. Experiments on the RSCUP competition dataset show that the proposed approach can effectively improve the segmentation performance.