Object detection of high-resolution optical remote sensing images is challenging due to two fundamental problems. One is the huge scale variation of objects in images, e.g., small vehicle and cross-sea bridge. The other one is the objects could take on arbitrary orientations because of the high angle shot. In this paper, we propose a Rotation and Scale-invariant Detector (RS-Det) for remote sensing images to solve the above problem in an unified network. Specifically, RS-Det consists of a deformable convolution module to learn spatial transformation (such as rotation, transition, etc) and a feature pyramid architecture for multi-scale feature representation. These two modules enable a better feature learning of convolutional neural network and boost the performance by 3.6% compared with the baseline method. In DOTA, a large-scale
dataset for aerial image object detection, our RS-Det achieves the state-of-the-art accuracy, which verifies our method’s superiority.