The accurate diagnosis of Alzheimer's disease (AD) has an important impact on early treatment. Positron emission tomography (PET) and magnetic resonance imaging (MRI) are popular imaging methods and are used to facilitate the identification and evaluation of AD. In this paper, we proposed a VGG-style 3D convolutional neural network (3D CNN) model, which is named 3D PET-MRI Net (3D PMNet), and it uses DiffGrad optimizer to speed up the convergence of the model and Focalloss function to improve the classification performance of unbalanced data processing. The multi-modal feature information of 3D MRI and PET images can be extracted using the 3D PMNet model, which provides convenience for AD diagnosis. Tenfold cross-validation was performed on the data of each patient in the data set to determine the group classification. The results showed that the proposed method achieves 97.49%, 81.25%, and 76.67% accuracy in the classification tasks of AD: NC, AD: MCI, and NC: MCI, respectively. Our PMNet reached 72.55% accuracy in AD: NC: MCI three group classification, which is significantly better than the other reported network models. © 2022 IEEE.