Cross-spectral image patch matching is still challenging due to significant nonlinear differences between image patches. Recently, image patch matching methods based on feature relation learning have attracted increasing attention and achieved good performance. However, we find that the metric learning methods based on feature difference cannot comprehensively and effectively extract useful discriminative information between image patch pairs by only adopting two branches network structure. Therefore, we propose a novel multi-branch feature difference learning network (MFD-Net). Specifically, we build a multi-branch parallel feature difference extraction network, which can capture richer and more discriminative feature difference information and achieve significant improvements on matching tasks. Furthermore, we propose a combined metric network composed of a master metric network module and multiple branch metric network modules, which promotes the forward update of network weights and reduces the similarity of features extracted by each feature difference extraction module with negligible increase in inference time. Extensive experimental results show that the proposed MFD-Net achieves superior performances on cross-spectral image patch matching and single spectral image patch matching.