Revisiting Confidence Estimation: Towards Reliable Failure Prediction
文献类型：期刊论文
| 作者 | Zhu, Fei1 ; Zhang, Xu-Yao2,3; Cheng, Zhen2,3; Liu, Cheng-Lin2,3
|
| 刊名 | IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
 |
| 出版日期 | 2024-05-01 |
| 卷号 | 46期号:5页码:3370-3387 |
| 关键词 | Confidence estimation uncertainty quantification failure prediction misclassification detection selective classification out-of-distribution detection confidence calibration model reliability trustworthy flat minima |
| ISSN号 | 0162-8828 |
| DOI | 10.1109/TPAMI.2023.3342285 |
| 通讯作者 | Zhang, Xu-Yao(xyz@nlpr.ia.ac.cn) |
| 英文摘要 | Reliable confidence estimation is a challenging yet fundamental requirement in many risk-sensitive applications. However, modern deep neural networks are often overconfident for their incorrect predictions, i.e., misclassified samples from known classes, and out-of-distribution (OOD) samples from unknown classes. In recent years, many confidence calibration and OOD detection methods have been developed. In this paper, we find a general, widely existing but actually-neglected phenomenon that most confidence estimation methods are harmful for detecting misclassification errors. We investigate this problem and reveal that popular calibration and OOD detection methods often lead to worse confidence separation between correctly classified and misclassified examples, making it difficult to decide whether to trust a prediction or not. Finally, we propose to enlarge the confidence gap by finding flat minima, which yields state-of-the-art failure prediction performance under various settings including balanced, long-tailed, and covariate-shift classification scenarios. Our study not only provides a strong baseline for reliable confidence estimation but also acts as a bridge between understanding calibration, OOD detection, and failure prediction. |
| 资助项目 | National Key Research and Development Program |
| WOS研究方向 | Computer Science ; Engineering |
| 语种 | 英语 |
| WOS记录号 | WOS:001196751500031 |
| 出版者 | IEEE COMPUTER SOC |
| 资助机构 | National Key Research and Development Program |
| 源URL | [http://ir.ia.ac.cn/handle/173211/58089]  |
| 专题 | 自动化研究所_模式识别国家重点实验室_模式分析与学习团队 |
| 通讯作者 | Zhang, Xu-Yao |
| 作者单位 | 1.Chinese Acad Sci, Hong Kong Inst Sci & Innovat, Ctr Artificial Intelligence & Robot, Hong Kong 999077, Peoples R China 2.Chinese Acad Sci, Inst Automat, State Key Lab Multimodal Artificial Intelligence S, Beijing 100190, Peoples R China 3.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100049, Peoples R China |
| 推荐引用方式 GB/T 7714 | Zhu, Fei,Zhang, Xu-Yao,Cheng, Zhen,et al. Revisiting Confidence Estimation: Towards Reliable Failure Prediction[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2024,46(5):3370-3387. |
| APA | Zhu, Fei,Zhang, Xu-Yao,Cheng, Zhen,&Liu, Cheng-Lin.(2024).Revisiting Confidence Estimation: Towards Reliable Failure Prediction.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,46(5),3370-3387. |
| MLA | Zhu, Fei,et al."Revisiting Confidence Estimation: Towards Reliable Failure Prediction".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 46.5(2024):3370-3387. |
入库方式： OAI收割
来源：自动化研究所
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。