Stripe noise removal is a crucial step for the infrared imaging system. Existing stripe removal methods are hard to balance stripe removal and image details preservation. In this paper, a deep multi-scale dense connection convolutional neural network (DMD-CNN) is proposed to address this problem. In DMD-CNN, a multi-scale feature representation unit (FR-Unit) is designed to decompose raw image into different scales which can extract diverse fine and coarse features. Dense connection is introduced into the network, which makes full use of the multi-scale information obtained by FR-Unit and avoids performance degradation. Moreover, the regularization term Lh is defined to depict the vertical direction smoothness property of stripe. Experiment results show that DMD-CNN performs more stable stripe removal effects in different scenes and diverse stripe intensity. Meanwhile, DMD-CNN outperforms seven state-of-the-art stripe removal methods on qualitative and quantitative evaluation.