Hyperspectral imagery contains spatial and spectral information, which can reveal the material properties of the target while intuitively displaying its spatial attributes. It has been applied in target recognition, search and rescue, and other fields. However, manual detection inevitably leads to missed detections and false alarms, necessitating the assistance of artificial intelligence for detection. To address this, we propose the multilevel-based algorithm for hyperspectral image interpretation. 1) From the spatial and spectral dimensions, we propose a semantic segmentation algorithm based on multidimensional information fusion to achieve semantic segmentation. 2) From the semantic and textual representation dimensions, we introduce a context interpretation module based on visual attention. We construct both real and simulated databases to validate the effectiveness of the algorithm. Experimental results demonstrate that the average accuracy of semantic segmentation achieved by the proposed algorithm is 74.3%. Additionally, the BLEU1 score reaches 71.2, outperforming mainstream algorithms by 1.4. © 2023 Elsevier Ltd