Recent single image deraining works have achieved significant improvement using convolutional neural networks. However, the rain streaks in the rain image share similar patterns with its multi-scale versions, which are not fully exploited in recent works. In this paper, we propose an Across-scale Progressive Attention Network (i.e., APAN) to explore the multi-scale collaborative representation for single image deraining. Specifically, we represent each rainy image via a multi-scale module. An across-scale attention module is then used to capture long-range feature correspondences from multi-scale features, which can model the rain streaks at an enlarging feature dimension. Afterwards, we construct a pyramid structure and further predict the rain streak progressively, which also guides the across-scale attention module to refine the feature representation from coarse to fine. The proposed model exploits self-similarity of features via an across-scale attention between different scales, which can well model the rain streak with long-range information. Experiments on several datasets show that our model achieves significant improvement compared with most state-of-the-art deraining models.