Although scene text recognition has achieved high performance, text segmentation still needs to be improved. The goal of text segmentation is to obtain pixel-level foreground text masks from scene images. In this paper, we adaptively resize the input images to their optimal scales and propose the Reffned Pyramid Feature Fusion Network (RPFF-Net) for robust scene text segmentation. To address the issue of inconsistent text scaling, we propose an adaptive image scaling method that takes into account the density of text regions in each scene image. In the RPFF-Net, we ffrst extract multi-scale features from the backbone network, and then combine these features using effective pyramid feature fusion methods. To enhance the interaction between texts from contextual characters and extract features at different levels, we apply two self-attention mechanisms to the fusion feature map in spatial and channel dimensions. The experimental results demonstrate the effectiveness of our approach on several text segmentation benchmarks including the monolingual TextSeg and bilingual BTS dataset, and show that it outperforms the existing state-of-the-art scene text segmentation methods even without OCR (optical character recognition) enhancement.