Few-shot video classification (video FSL) has gained increasing attention in recent years, which learns classifiers for novel concepts from only a few samples. The existing methods rarely consider the local-global relation for video feature learning, which would ultimately result in low discriminative ability. Recently, Capsule Network (CapsNet) has shown significant potential in local-global relation learning in image analysis field. However, CapsNet cannot be directly applied in video FSL, since it ignores the interaction between videos and has high computational complexity. In this paper, a Dual-Routing Capsule Graph Neural Network (DR-CapsGNN) is proposed to solve the above issues. The DR-CapsGNN leverages CapsNet and Graph Neural Network (GNN) to explore local-global relation and preserve the detailed properties. Specifically, the CapsGNN is used to learn video relation and structural information to generate high-quality hierarchical capsules. Furthermore, a novel dual-routing mechanism is designed to filter low-discriminative capsules from a holistic perspective and achieves high efficiency, which consists of inter-video routing and intra-video routing. Extensive experimental results demonstrate that our proposed approach performs favorably against state-of-the-art methods on two popular benchmarks.