Mild cognitive impairment (MCI) is the preliminary stage of dementia, and has a high risk of progression to Alzheimer's disease (AD) in the elderly. Early detection of MCI plays a vital role in preventing progression of AD. Clinical diagnosis of MCI requires many examinations, which are highly demanding on hospital equipment and expensive for patients. Electroencephalography (EEG) offers a non-invasive and less expensive way to diagnose MCI early. In this paper, we propose a multi-modal fusion classification framework for MCI detection. We collect EEG data using a delayed match-to-sample task and analyze the differences between the two groups. Based on analysis results, we extract Power spectral density (PSD), PSD enhanced, Event-related potential (ERP) features in EEG signal along with physiological features and behavioral features of the subjects to classify MCI and healthy elderly. By comparing the impact of different features on classification performance, we find that the time-domain based ERP features are better than the frequency-domain based PSD or PSD enhanced features to overcome inter-individual differences to distinguish MCI, and these two features have good complementarity, fusing ERP and PSD enhanced features can greatly improve the classification accuracy to 84.74%. The final result shows that MCI and healthy elderly can be well classified by using this framework.