Motion prediction of nearby traffic agents is critical for autonomous driving. It is obvious that traffic agents’ trajectories are influenced by physical lane rules and agents’ social interactions. In this paper, we propose the social relation and physical lane aggregator for multimodal motion prediction, where the social relations of agents are mainly captured with graph convolutional networks and self-attention mechanism, and then fused with the physical lane via the self-attention mechanism. The proposed methods are evaluated on the Waymo Open Motion Dataset, and the results show the effectiveness of the proposed two feature aggregation modules for trajectory prediction.