Structured Neural Decoding With Multitask Transfer Learning of Deep Neural Network Representations
文献类型：期刊论文
| 作者 | Du, Changde2,3,4 ; Du, Changying1; Huang, Lijie4; Wang, Haibao3,4; He, Huiguang3,4,5
|
| 刊名 | IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
 |
| 出版日期 | 2022-02-01 |
| 卷号 | 33期号:2页码:600-614 |
| 关键词 | Decoding Image reconstruction Functional magnetic resonance imaging Visualization Task analysis Brain Correlation Deep neural network (DNN) functional magnetic resonance imaging (fMRI) image reconstruction multioutput regression neural decoding |
| ISSN号 | 2162-237X |
| DOI | 10.1109/TNNLS.2020.3028167 |
| 通讯作者 | He, Huiguang(huiguang.he@ia.ac.cn) |
| 英文摘要 | The reconstruction of visual information from human brain activity is a very important research topic in brain decoding. Existing methods ignore the structural information underlying the brain activities and the visual features, which severely limits their performance and interpretability. Here, we propose a hierarchically structured neural decoding framework by using multitask transfer learning of deep neural network (DNN) representations and a matrix-variate Gaussian prior. Our framework consists of two stages, Voxel2Unit and Unit2Pixel. In Voxel2Unit, we decode the functional magnetic resonance imaging (fMRI) data to the intermediate features of a pretrained convolutional neural network (CNN). In Unit2Pixel, we further invert the predicted CNN features back to the visual images. Matrix-variate Gaussian prior allows us to take into account the structures between feature dimensions and between regression tasks, which are useful for improving decoding effectiveness and interpretability. This is in contrast with the existing single-output regression models that usually ignore these structures. We conduct extensive experiments on two real-world fMRI data sets, and the results show that our method can predict CNN features more accurately and reconstruct the perceived natural images and faces with higher quality. |
| WOS关键词 | NATURAL IMAGES ; BRAIN ; RECONSTRUCTION ; FACES |
| 资助项目 | National Natural Science Foundation of China[61976209] ; National Natural Science Foundation of China[62020106015] ; National Natural Science Foundation of China[61906188] ; National Natural Science Foundation of China[61602449] ; Chinese Academy of Sciences (CAS) International Collaboration Key Project[173211KYSB20190024] ; CAS[XDB32040000] |
| WOS研究方向 | Computer Science ; Engineering |
| 语种 | 英语 |
| WOS记录号 | WOS:000752016400015 |
| 出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
| 资助机构 | National Natural Science Foundation of China ; Chinese Academy of Sciences (CAS) International Collaboration Key Project ; CAS |
| 源URL | [http://ir.ia.ac.cn/handle/173211/47361]  |
| 专题 | 类脑智能研究中心_神经计算及脑机交互 |
| 通讯作者 | He, Huiguang |
| 作者单位 | 1.Huawei Noahs Ark Lab, Beijing 100085, Peoples R China 2.Huawei Cloud BU EI Innovat Lab, Beijing 100085, Peoples R China 3.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100049, Peoples R China 4.Chinese Acad Sci, Res Ctr Brain Inspired Intelligence, Natl Lab Pattern Recognit, Inst Automat, Beijing 100190, Peoples R China 5.Chinese Acad Sci, Ctr Excellence Brain Sci & Intelligence Technol, Beijing 100190, Peoples R China |
| 推荐引用方式 GB/T 7714 | Du, Changde,Du, Changying,Huang, Lijie,et al. Structured Neural Decoding With Multitask Transfer Learning of Deep Neural Network Representations[J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,2022,33(2):600-614. |
| APA | Du, Changde,Du, Changying,Huang, Lijie,Wang, Haibao,&He, Huiguang.(2022).Structured Neural Decoding With Multitask Transfer Learning of Deep Neural Network Representations.IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,33(2),600-614. |
| MLA | Du, Changde,et al."Structured Neural Decoding With Multitask Transfer Learning of Deep Neural Network Representations".IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 33.2(2022):600-614. |
入库方式： OAI收割
来源：自动化研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。