中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 自动化研究所
  3. 中国科学院自动化研究所
  4. 多模态人工智能系统全国重点实验室
  5. 先进控制与自动化团队
Pay attention to doctor & ndash;patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis

文献类型：期刊论文

作者Zheng, Wenbo1,2; Yan, Lan2,3; Gou, Chao4; Zhang, Zhi-Cheng5; Zhang, Jun Jason6; Hu, Ming7; Wang, Fei-Yue2
刊名INFORMATION FUSION
出版日期2021-11-01
卷号75页码:168-185
关键词COVID-19 diagnose Knowledge attention mechanism Knowledge-based representation learning Knowledge embedding
ISSN号1566-2535
DOI10.1016/j.inffus.2021.05.015
通讯作者Wang, Fei-Yue(feiyue.wang@ia.ac.cn)
英文摘要The sudden increase in coronavirus disease 2019 (COVID-19) cases puts high pressure on healthcare services worldwide. At this stage, fast, accurate, and early clinical assessment of the disease severity is vital. In general, there are two issues to overcome: (1) Current deep learning-based works suffer from multimodal data adequacy issues; (2) In this scenario, multimodal (e.g., text, image) information should be taken into account together to make accurate inferences. To address these challenges, we propose a multi-modal knowledge graph attention embedding for COVID-19 diagnosis. Our method not only learns the relational embedding from nodes in a constituted knowledge graph but also has access to medical knowledge, aiming at improving the performance of the classifier through the mechanism of medical knowledge attention. The experimental results show that our approach significantly improves classification performance compared to other state-of-the-art techniques and possesses robustness for each modality from multi-modal data. Moreover, we construct a new COVID-19 multi-modal dataset based on text mining, consisting of 1393 doctor-patient dialogues and their 3706 images (347 X-ray + 2598 CT + 761 ultrasound) about COVID-19 patients and 607 non-COVID-19 patient dialogues and their 10754 images (9658 X-ray + 494 CT + 761 ultrasound), and the fine-grained labels of all. We hope this work can provide insights to the researchers working in this area to shift the attention from only medical images to the doctor-patient dialogue and its corresponding medical images.
WOS关键词PREDICTING COVID-19 ; FUSION ; ACCURATE ; NETWORK ; CANCER ; SYSTEM
资助项目National Key R&D Program of China[2020YFB1600400] ; National Natural Science Foundation of China[61806198] ; National Natural Science Foundation of China[61533019] ; National Natural Science Foundation of China[U1811463] ; Key Technologies Research and Development Program of Guangzhou, China[202007050002] ; National Key Research and Development Program of China[2018AAA0101502]
WOS研究方向Computer Science
语种英语
WOS记录号WOS:000671018300014
出版者ELSEVIER
资助机构National Key R&D Program of China ; National Natural Science Foundation of China ; Key Technologies Research and Development Program of Guangzhou, China ; National Key Research and Development Program of China
源URL[http://ir.ia.ac.cn/handle/173211/45643]  
专题自动化研究所_复杂系统管理与控制国家重点实验室_先进控制与自动化团队
通讯作者Wang, Fei-Yue
作者单位1.Xi An Jiao Tong Univ, Sch Software Engn, Xian 710049, Peoples R China
2.Chinese Acad Sci, Inst Automat, State Key Lab Management & Control Complex Syst, Beijing 100190, Peoples R China
3.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100190, Peoples R China
4.Sun Yat Sen Univ, Sch Intelligent Syst Engn, Guangzhou 510275, Peoples R China
5.Gen Hosp Peoples Liberat Army, Med Ctr 7, Beijing 100700, Peoples R China
6.Wuhan Univ, Sch Elect Engn & Automat, Wuhan 430072, Peoples R China
7.Wuhan Pulm Hosp, Intens Care Unit, Wuhan 430030, Peoples R China
推荐引用方式
GB/T 7714		 Zheng, Wenbo,Yan, Lan,Gou, Chao,et al. Pay attention to doctor & ndash;patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis[J]. INFORMATION FUSION,2021,75:168-185.
APA Zheng, Wenbo.,Yan, Lan.,Gou, Chao.,Zhang, Zhi-Cheng.,Zhang, Jun Jason.,...&Wang, Fei-Yue.(2021).Pay attention to doctor & ndash;patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis.INFORMATION FUSION,75,168-185.
MLA Zheng, Wenbo,et al."Pay attention to doctor & ndash;patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis".INFORMATION FUSION 75(2021):168-185.

入库方式： OAI收割

来源：自动化研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。