Accurate and efficient image segmentation can contribute to improving the recognition rate of surface defects for hot-rolled steel strips. However, due to its variances in shape, position, defect type and fuzzy boundary, surface defect segmentation is a challenging task. To address this issue, a depth-wise separable U-shape network (DSUNet) is proposed. In order to reduce the computation complexity and accelerate the segmentation performance, depth-wise separable convolution is employed to replace the traditional convolutional layer. In addition, a multi-scale module is proposed to extract multi-scale context and improve the segmentation accuracy. The experimental results indicate that the accuracy and dice of DSUNet reach 95.42% and 80.8%, respectively, and the DSUNet can segment 38.5 images per second, which suggests that the DSUNet can precisely segment surface defects for hot-rolled steel strip with high efficiency.