The goal of this paper is to simultaneously segment the object regions in a set of images with the same object class, known as object co-segmentation. Different from typical methods, simply assuming that the common regions among images are the object regions, we additionally consider the disturbance from consistent backgrounds, and indicate not only common regions but salient ones among images to be the object regions. To this end, bwe propose an Adaptive Discriminative Low Rank matrix Recovery (ADLRR) algorithm to divide the over-completely segmented regions (i.e., super-pixels) of a given image set into object and non-object ones. The proposed ADLRR is formulated from two views: a low-rank matrix recovery term for salient regions detection and a discriminative learning term  adopted to distinguish object regions from all super-pixels. An additional regularized term is incorporated to jointly measure the disagreement between the predicted saliency and the objectiveness probability. For the unified learning problem by connecting the above three terms, we design an efficient alternate optimization procedure based on block-coordinate descent and augmented Lagrange multipliers method. Extensive experiments are conducted on three public datasets, i.e., MSRC, iCoseg and Caltech101, and the comparisons with some state-of-the-arts demonstrate the effectiveness of our work.