More than 95% of colorectal cancers are gradually transformed from polyps, so regular colonoscopy polyp examination plays an important role in cancer prevention and early treatment. However, automatic polyp segmentation remains a challenging task due to the low-contrast tissue environment and the small size and variety (e.g., shape, color, texture) of polyps. In this case, the rich context information in colonoscopy images is worth exploring to address the above issues. On the one hand, the image-level context with a global receptive field can be used to enhance the discrimination between the foreground and the background to alleviate the occult and indistinguishability of polyps in colonoscopy images. On the other hand, the surrounding-level context focused on the surrounding pathological region of the polyp has more detailed features that are beneficial for polyp segmentation. Therefore, we propose a novel network named ISCNet that aims to fuse image-level and surrounding-level context information for polyp segmentation. Specifically, we first introduce the Global-Guided Context Aggregation (GGCA) module to explicitly model the foreground and background of polyp segmentation through image-level context, thereby flexibly enhancing polyp-related features and suppressing background-related features. Then, we design the Diverse Surrounding Context Focus (DSCF) module to focus on the surrounding area of the polyp to extract diverse local contexts to refine the segmentation results. Finally, we fuse the feature maps derived from these two modules so that our ISCNet can enjoy the facilitation of both the image-level and surrounding-level context information. To verify the effectiveness of our method, we conduct comprehensive experimental evaluations on three challenging datasets. The quantitative and qualitative experimental results confirm that our ISCNet outperforms current state-of-the-art methods by a large margin. Our code is available at https://github.com/vvmedical/ISCNet.