To precisely diagnose the brain tumor types and grades, magnetic resonance imaging (MRI), which is a kind of multisequence imaging technology, is usually applied. However, with the limitations of databases, most current computer-aided brain tumor diagnosis methods employ only a single MRI sequence, and the generalizability of these methods is not ideal. To improve the brain tumor diagnosis performance, an adaptive multisequence fusing neural network (AMF-Net), which can merge the characteristics of different MRI sequences with adaptive weights, is proposed. Inspired by the approximate horizontal symmetry of brains and manual diagnosis process, normalized horizontal differential images are adopted as the spatial attention mechanism, and dense skip connections from T2-weighted (T2-W) sequences are implemented to emphasize the importance of the T2-W sequences. Moreover, to adaptively combine different MRI sequences, an innovative self-learning mechanism, namely adaptive sequence fusion (ASF) module, is proposed. The experimental results show that the average accuracies of the AMF-Net on two databases reach 98.1% and 92.1%, respectively, and the application of the proposed spatial attention mechanism and the ASF module can improve the average accuracy on two databases by 1.7%/1.7% and 1.3%/2.1%, respectively, which indicates that the proposed spatial attention mechanism and the ASF module can improve the performance for brain tumor diagnosis tasks.