Emotion Knowledge Driven Video Highlight Detection
文献类型：期刊论文
| 作者 | Qi, Fan1,4; Yang, Xiaoshan1,2,3 ; Xu, Changsheng1,2,3
|
| 刊名 | IEEE TRANSACTIONS ON MULTIMEDIA
 |
| 出版日期 | 2021 |
| 卷号 | 23页码:3999-4013 |
| 关键词 | Visualization Training data Predictive models Training Semantics Emotion recognition Computational modeling Deep ranking knowledge graph video highlight detection |
| ISSN号 | 1520-9210 |
| DOI | 10.1109/TMM.2020.3035285 |
| 通讯作者 | Xu, Changsheng(csxu@nlpr.ia.ac.cn) |
| 英文摘要 | This paper addresses video highlight detection which aims to select a small subset of frames according to user's major or special interest. The performances of conventional methods highly depend on large-scale manually labeled training data which are time-consuming and labor-intensive to collect. To deal with this problem, we trace back to the original problem definition and find that whether a user is interested in a specific video segment heavily depends on human's subjective emotions. Leveraging this insight, we introduce an emotion knowledge driven video detection framework for modeling human's general emotion and inferencing highlight strength. Firstly, we obtain the concept-level representation of the video clip with a front-end network. The concepts are used as nodes to build an emotion-related knowledge graph, and their relationships in the graph are modeled via external public knowledge graphs. Then we adopt Siamese GCNs to model the dependencies between nodes in the graph and propagate messages along the edges. Finally, we compute the emotion-aware representation of the video clip based on the GCN layers and further use it to predict the highlight score. Our framework, including the front-end network, graph convolution layers and the highlight mapping network, can be trained in an end-to-end manner with the constraint of a ranking loss. Experiments on two benchmark datasets show that our proposed method performs favorably against the state-of-the-art methods. |
| WOS关键词 | RETRIEVAL |
| 资助项目 | National Key Research and Development Program of China[2018AAA0100604] ; National Natural Science Foundation of China[61720106006] ; National Natural Science Foundation of China[62072455] ; National Natural Science Foundation of China[61702511] ; National Natural Science Foundation of China[61751211] ; National Natural Science Foundation of China[61620106003] ; National Natural Science Foundation of China[61532009] ; National Natural Science Foundation of China[U1836220] ; National Natural Science Foundation of China[U1705262] ; National Natural Science Foundation of China[61872424] ; Key Research Program of Frontier Sciences of CAS[QYZDJSSWJSC039] ; Research Program of National Laboratory of Pattern Recognition[Z-2018007] |
| WOS研究方向 | Computer Science ; Telecommunications |
| 语种 | 英语 |
| WOS记录号 | WOS:000720519900007 |
| 出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
| 资助机构 | National Key Research and Development Program of China ; National Natural Science Foundation of China ; Key Research Program of Frontier Sciences of CAS ; Research Program of National Laboratory of Pattern Recognition |
| 源URL | [http://ir.ia.ac.cn/handle/173211/46455]  |
| 专题 | 自动化研究所_模式识别国家重点实验室_多媒体计算与图形学团队 |
| 通讯作者 | Xu, Changsheng |
| 作者单位 | 1.Peng Cheng Lab, Shenzhen 518055, Peoples R China 2.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100190, Peoples R China 3.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China 4.HeFei Univ Technol, Comp Sci, Hefei 230009, Peoples R China |
| 推荐引用方式 GB/T 7714 | Qi, Fan,Yang, Xiaoshan,Xu, Changsheng. Emotion Knowledge Driven Video Highlight Detection[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2021,23:3999-4013. |
| APA | Qi, Fan,Yang, Xiaoshan,&Xu, Changsheng.(2021).Emotion Knowledge Driven Video Highlight Detection.IEEE TRANSACTIONS ON MULTIMEDIA,23,3999-4013. |
| MLA | Qi, Fan,et al."Emotion Knowledge Driven Video Highlight Detection".IEEE TRANSACTIONS ON MULTIMEDIA 23(2021):3999-4013. |
入库方式： OAI收割
来源：自动化研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。