Distant supervision is an effective way to collect large-scale training data for relation extraction. To better solve the wrong labeling problem accompanied by distant supervision, some methods have been proposed to remove noise sentences directly. However, these methods seldom consider the relation label when removing noise sentences, neglecting the fact that a sentence is regarded as noise because the relation it expresses is inconsistent with the relation label. In this paper, we propose a novel method to improve the performance of bag-level relation extractor via removing noise data with a sentence selector. Specifically, the gated convolutional unit of the sentence selector can selectively output features related to the given relation, and these features will be used to judge whether a sentence expresses the given relation. The sentence selector is trained with the data automatically labeled by the relation extractor, and the relation extractor improves its performance with the highquality data selected by the sentence selector. These two modules are trained alternately, and both of them have achieved better performance. Experimental results show that our model significantly improves the performance of the relation extractor and outperforms competitive baseline methods.