中国科学院机构知识库网格
Chinese Academy of Sciences Institutional Repositories Grid
首页 机构 成果 学者
  
  1. CAS IR Grid
  2. 自动化研究所
  3. 中国科学院自动化研究所
  4. 多模态人工智能系统全国重点实验室
  5. 多媒体计算与图形学团队
Cross-Modal Federated Human Activity Recognition

文献类型：期刊论文

作者Yang, Xiaoshan1,2,3; Xiong, Baochen1,2,3; Huang, Yi1,2; Xu, Changsheng1,2,3
刊名IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
出版日期2024-08-01
卷号46期号:8页码:5345-5361
关键词Cross-modal learning federated learning human activity recognition
ISSN号0162-8828
DOI10.1109/TPAMI.2024.3367412
通讯作者Xu, Changsheng(csxu@nlpr.ia.ac.cn)
英文摘要Federated human activity recognition (FHAR) has attracted much attention due to its great potential in privacy protection. Existing FHAR methods can collaboratively learn a global activity recognition model based on unimodal or multimodal data distributed on different local clients. However, it is still questionable whether existing methods can work well in a more common scenario where local data are from different modalities, e.g., some local clients may provide motion signals while others can only provide visual data. In this article, we study a new problem of cross-modal federated human activity recognition (CM-FHAR), which is conducive to promote the large-scale use of the HAR model on more local devices. CM-FHAR has at least three dedicated challenges: 1) distributive common cross-modal feature learning, 2) modality-dependent discriminate feature learning, 3) modality imbalance issue. To address these challenges, we propose a modality-collaborative activity recognition network (MCARN), which can comprehensively learn a global activity classifier shared across all clients and multiple modality-dependent private activity classifiers. To produce modality-agnostic and modality-specific features, we learn an altruistic encoder and an egocentric encoder under the constraint of a separation loss and an adversarial modality discriminator collaboratively learned in hyper-sphere. To address the modality imbalance issue, we propose an angular margin adjustment scheme to improve the modality discriminator on modality-imbalanced data by enhancing the intra-modality compactness of the dominant modality and increase the inter-modality discrepancy. Moreover, we propose a relation-aware global-local calibration mechanism to constrain class-level pairwise relationships for the parameters of the private classifier. Finally, through decentralized optimization with alternative steps of adversarial local updating and modality-aware global aggregation, the proposed MCARN obtains state-of-the-art performance on both modality-balanced and modality-imbalanced data.
资助项目National Natural Science Foundation of China[62036012] ; National Natural Science Foundation of China[U23A20387] ; National Natural Science Foundation of China[62322212] ; National Natural Science Foundation of China[62072455]
WOS研究方向Computer Science ; Engineering
语种英语
WOS记录号WOS:001262841000026
出版者IEEE COMPUTER SOC
资助机构National Natural Science Foundation of China
源URL[http://ir.ia.ac.cn/handle/173211/59216]  
专题自动化研究所_模式识别国家重点实验室_多媒体计算与图形学团队
通讯作者Xu, Changsheng
作者单位1.Chinese Acad Sci, Inst Automat, State Key Lab Multimodal Artificial Intelligence, Beijing 100190, Peoples R China
2.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100190, Peoples R China
3.Peng Cheng Lab, Shenzhen 518055, Peoples R China
推荐引用方式
GB/T 7714		 Yang, Xiaoshan,Xiong, Baochen,Huang, Yi,et al. Cross-Modal Federated Human Activity Recognition[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2024,46(8):5345-5361.
APA Yang, Xiaoshan,Xiong, Baochen,Huang, Yi,&Xu, Changsheng.(2024).Cross-Modal Federated Human Activity Recognition.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,46(8),5345-5361.
MLA Yang, Xiaoshan,et al."Cross-Modal Federated Human Activity Recognition".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 46.8(2024):5345-5361.

入库方式： OAI收割

来源：自动化研究所

浏览0
下载0
收藏0
其他版本

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。