The grain size is an important steel grading parameter. For metallographic steel images with various grain sizes and complex textures, it is not possible for a human expert to determine the grain size efficiently. Meanwhile, conventional computer vision models are designed based on general images and they are not capable of achieving high performance in metallographic steel grain size recognition. To solve these problems, a method based on multiple receptive field fusion is proposed. A multi-scale convolutional net is used to extract information of microstructures in various scales. In addition, to augment the extracted features, a self-attention module is used to improve the robustness of feature representation with complex metallographic textures. At last, via a multiple feature fusion module, the data capacity is extended by projecting features into multiple hidden spaces. A comprehensive experiment was conducted on the Huawei Cloud Dataset and the classification accuracy was improved by 27% compared with other SOTA models, while our computation cost was only 0.06 GFLOPs