Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable Cancer Survival Analysis
文献类型：期刊论文
| 作者 | Yan, Rui2,3; Zhang, Xueyuan4; Jiang, Zihang2,3; Wang, Baizhi2,3; Bian, Xiuwu5,6; Ren, Fei7; Zhou, S. Kevin1,2,3,8 |
| 刊名 | IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
 |
| 出版日期 | 2026 |
| 卷号 | 48期号:1页码:896-913 |
| 关键词 | Pathology Cancer Transformers Feature extraction Data models Biological system modeling Analytical models Deep learning Semantics Multimodal transformer model interpretability survival analysis pathological image analysis gene expression gene expression |
| ISSN号 | 0162-8828 |
| DOI | 10.1109/TPAMI.2025.3611531 |
| 英文摘要 | Integrating multimodal data of pathological image and gene expression for cancer survival analysis can achieve better results than using a single modality. However, existing multimodal learning methods ignore fine-grained interactions between both modalities, especially the interactions between biological pathways and pathological image patches. In this article, we propose a novel Pathway-Aware Multimodal Transformer (PAMT) framework for interpretable cancer survival analysis. Specifically, the PAMT learns fine-grained modality interaction through three stages: (1) In the intra-modal pathway-pathway / patch-patch interaction stage, we use the Transformer model to perform intra-modal information interaction; (2) In the inter-modal pathway-patch alignment stage, we introduce a novel label-free contrastive loss to aligns semantic information between different modalities so that the features of the two modalities are mapped to the same semantic space; and (3) In the inter-modal pathway-patch fusion stage, to model the medical prior knowledge of "genotype determines phenotype", we propose a pathway-to-patch cross fusion module to perform inter-modal information interaction under the guidance of pathway prior. In addition, the inter-modal cross fusion module of PAMT endows good interpretability, helping a pathologist to screen which pathway plays a key role, to locate where on whole slide image (WSI) are affected by the pathway, and to mine prognosis-relevant pathology image patterns. Experimental results based on three datasets of bladder urothelial carcinoma, lung squamous cell carcinoma, and lung adenocarcinoma demonstrate that the proposed framework significantly outperforms the state-of-the-art methods. |
| 资助项目 | Science and Technology Innovation Key R&D Program of Chongqing[CSTB2022TIAD-STX0008] ; Jiangsu Funding Program for Excellent Postdoctoral Talent[2024ZB841] ; National Natural Science Foundation of China[62402473] ; National Natural Science Foundation of China[62271465] ; Suzhou Basic Research Program[SYG202338] |
| WOS研究方向 | Computer Science ; Engineering |
| 语种 | 英语 |
| WOS记录号 | WOS:001630282400001 |
| 出版者 | IEEE COMPUTER SOC |
| 源URL | [http://119.78.100.204/handle/2XEOYT63/42937]  |
| 专题 | 中国科学院计算技术研究所 |
| 通讯作者 | Ren, Fei; Zhou, S. Kevin |
| 作者单位 | 1.Jiangsu Prov Key Lab Multimodal Digital Twin Techn, Suzhou 215123, Peoples R China 2.Univ Sci & Technol China USTC, Sch Biomed Engn, Hefei 230026, Peoples R China 3.USTC, Ctr Med Imaging, Robot Analyt Comp & Learning MIRACLE, Suzhou Inst Adv Res, Suzhou 215123, Peoples R China 4.Chongqing Zhijian Life Technol Co Ltd, Chongqing 400039, Peoples R China 5.Army Med Univ, Southwest Hosp, Key Lab Tumor Immunopathol,Inst Pathol, Minist Educ China,Third Mil Med Univ, Chongqing 400039, Peoples R China 6.Southwest Hosp, Southwest Canc Ctr, Key Lab Tumor Immunopathol, Minist Educ China, Chongqing 400039, Peoples R China 7.Chinese Acad Sci, Inst Comp Technol, State Key Lab Proc, Beijing 100190, Peoples R China 8.Univ Sci & Technol China, State Key Lab Precis & Intelligent Chem, Hefei 230026, Peoples R China |
| 推荐引用方式 GB/T 7714 | Yan, Rui,Zhang, Xueyuan,Jiang, Zihang,et al. Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable Cancer Survival Analysis[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2026,48(1):896-913. |
| APA | Yan, Rui.,Zhang, Xueyuan.,Jiang, Zihang.,Wang, Baizhi.,Bian, Xiuwu.,...&Zhou, S. Kevin.(2026).Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable Cancer Survival Analysis.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,48(1),896-913. |
| MLA | Yan, Rui,et al."Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable Cancer Survival Analysis".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 48.1(2026):896-913. |
入库方式： OAI收割
来源：计算技术研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。