Emotion recognition is a major research point in the field of affective computing. Existing research on the application of physiological signals to emotion recognition mainly focuses on the processing of contact signals. However, there are issues with contact signal acquisition equipment, such as limited portability and poor user compliance, which make it difficult to promote its use. To explore a new method for emotion recognition based on contactless ballistocardiogram (BCG), we proposed a SE-CNN model with a multi-class focal loss function. To construct the dataset, we used audio-visual stimuli to evoke the subjects' emotions and collected data on the subjects' three discrete emotions, positive, neutral, and negative, through our established BCG signal acquisition system based on a piezoelectric ceramics sensor. Root mean square filter and thresholding were used to detect and eliminate motion artifacts of BCG signals. We did two kinds of preprocessing on BCG signals: wavelet transform and bandpass filtering, to explore the effect of different components of BCG on emotion recognition. Subsequently, we verified the model's performance and cross-time working ability through traditional K-Fold and our proposed K-Session cross-validation methods. The results showed that the band-pass filtering method was more beneficial to the current classification task. Under K-Fold cross-validation, the model's accuracy, precision, and recall were 97.21%, 97.00%, and 97.11%. Under K-Session cross-validation, the model's accuracy, precision, and recall were 94.66%, 93.92%, and 94.86%, respectively, all of which were better than the classification effect of synchronous ECG. The reliability of BCG in contactless emotion recognition was proved.