Reading irregular text of arbitrary shape or low quality in natural scene images is a challenging task. Existing irregular scene text recognition methods mainly focus on irregular text with arbitrary shape, but rarely focus on irregular text of low quality. In this work, we propose a simple but effective method for recognizing irregular texts with arbitrary shape and low quality simultaneously. The proposed Multi-Branch guided Attention Network (MBAN) makes mutual guidance among multi-branch data in training, so as to learn invariant semantic representation between regular text images and the corresponding irregular images. Compared with the standard attention framework for text recognition, MBAN can significantly improve the performance of irregular text recognition while preserving similar performance for regular text recognition. In addition, regarding the attention drift problem encountered in standard attention network, MBAN can significantly improve the accuracy of alignment factors at each time step. We verify the effectiveness of MBAN in irregular text recognition and attention drift problem through extensive experiments. The performance of MBAN is shown to be comparable on regular datasets and superior on some irregular datasets with state-of-the-art methods.