In the past decade, deep learning methods have been implemented in the medical image fields and have achieved good performance. Recently, deep learning algorithms have been successful in the evaluation of diagnosis on lung images. Although chest radiography (CR) is the standard data modality for diagnosing pneumoconiosis, computed tomography (CT) typically provides more details of the lesions in the lung. Thus, a transformer-based factorized encoder (TBFE) was proposed and first applied for the classification of pneumoconiosis depicted on 3D CT images. Specifically, a factorized encoder consists of two transformer encoders. The first transformer encoder enables the interaction of intra-slice by encoding feature maps from the same slice of CT. The second transformer encoder explores the inter-slice interaction by encoding feature maps from different slices. In addition, the lack of grading standards on CT for labeling the pneumoconiosis lesions. Thus, an acknowledged CR-based grading system was applied to mark the corresponding pneumoconiosis CT stage. Then, we pre-trained the 3D convolutional autoencoder on the public LIDC-IDRI dataset and fixed the parameters of the last convolutional layer of the encoder to extract CT feature maps with underlying spatial structural information from our 3D CT dataset. Experimental results demonstrated the superiority of the TBFE over other 3D-CNN networks, achieving an accuracy of 97.06%, a recall of 89.33%, precision of 90%, and an F1-score of 93.33%, using 10-fold cross-validation.