Alzheimer's disease (AD) is a chronic neurodegenerative disease. Brain atrophy is one of the most common hallmarks of AD, which can be accessed by structural magnetic resonance imaging (sMRI). Recent advances in deep learning models based on convolutional neural networks (CNN) have led to excellent results in classifying AD from normal controls (NC). Inspired by the concept of brain connections, we proposed a 3D Nested Hierarchical Transformer (3DNesT) model featuring a hierarchical local self-attention mechanism for diagnosing AD and provided the interpretation of the model. We calculated the attention of regions within each single block and merged the blocks hierarchically to integrate the information throughout the brain for diagnosis. Using sMRI from 1159 NC and 833 AD from three multisite datasets, the proposed model achieves 90.06±2.26% accuracy in 10-fold cross-validation and 82.70%, 90.08%, and 91.60% in inter-site cross-validation. Our model has more trainable parameters and achieves better accuracy with less memory cost than the traditional CNN model. The model puts more attention in the medial temporal lobe when identifying NC, while the attention is spread out when predicting AD. Furthermore, the predicted values were significantly correlated with cognitive scores and key neurobiological indicators in AD, demonstrating the clinical validity of the 3DNesT model.