Self-supervised learning (SSL) holds promise in leveraging large amounts of unlabeled data. However, the success of popular SSL methods has limited on single-centricobject images like those in ImageNet and ignores the correlation among the scene and instances, as well as the semantic difference of instances in the scene. To address the
above problems, we propose a Unified Self-supervised Visual Pre-training (UniVIP), a novel self-supervised framework to learn versatile visual representations on either
single-centric-object or non-iconic dataset. The framework takes into account the representation learning at three
levels: 1) the similarity of scene-scene, 2) the correlation of scene-instance, 3) the discrimination of instanceinstance. During the learning, we adopt the optimal transport algorithm to automatically measure the discrimination of instances. Massive experiments show that UniVIP pre-trained on non-iconic COCO achieves state-ofthe-art transfer performance on a variety of downstream
tasks, such as image classification, semi-supervised learning, object detection and segmentation. Furthermore, our
method can also exploit single-centric-object dataset such
as ImageNet and outperforms BYOL by 2.5% with the same
pre-training epochs in linear probing, and surpass current
self-supervised object detection methods on COCO dataset,
demonstrating its universality and potential.