Deep Learning (DL) is capable of excavating features hidden deep in complex data. In this paper, we introduce hierarchical convolutional neural networks (HCNN) to implement the EEG-based emotion classifier (positive, negative and neutral) in a movie-watching task. Differential Entropy (DE) is calculated as features at certain time interval for each channel. We organize features from different channels into two dimensional maps to train HCNN classifier. This approach extracts features contained in the spatial topology of electrodes directly, which is often neglected by the widely-used one-dimensional models. The performance of HCNN was compared with one-dimensional deep model SAE (Stacked Autoencoder), as well as traditional shallow models SVM and KNN. We find that HCNN (88.2% ± 3.5%) is better than SAE (85.4% ± 8.1%), and deep models are more favorable in emotion recognition BCI (Brain-computer Interface) system than shallow models. Moreover, we show that models learned on one person is hard to transfer to others and the individual difference in EEG emotion-related signal is significant among peoples. Finally, we find Beta and Gamma (rather than Delta, Theta and Alpha) waves play the key role in emotion recognition.