Brain signal-based affective computing has recently drawn considerable attention due to its potential widespread applications. Most existing efforts exploit emotion similarities or brain region similarities to learn emotion representations. However, the relationships between emotions and brain regions are not explicitly incorporated into the representation learning process. Consequently, the learned representations may not be informative enough to benefit downstream tasks, e.g., emotion decoding. In this work, we propose a novel neural decoding framework, Graph Emotion Decoding (GED), which integrates the relationships between emotions and brain regions via a bipartite graph structure into the neural decoding process. Further analysis shows that exploiting such relationships helps learn better representations, verifying the rationality and effectiveness of GED. Comprehensive experiments on visually evoked emotion datasets demonstrate the superiority of our model.