Due to more abundant data sources, more various objects of interest, and more time-consuming annotations, there is a large amount of out-of-distribution (OOD) data in the remote sensing field, on which the performance of high-accuracy image segmentation models trained under ideal experimental conditions generally degrades dramatically. Domain Adaptation (DA) consequently comes into being, which aims to learn the predictor for the label-scarce target domain of interest with the help of the label-sufficient source domain in the presence of the distribution difference, namely domain shift, between the two domains. However, the off-the-shelf DA methods for image segmentation not only struggle to cope with the more complex domain shift problems in remote sensing imagery, but also almost cannot process heterogeneous data directly without information loss. While the current heterogeneous DA methods mostly still rely on some supervision information from the target domain, which is typically inaccessible in the real world. To overcome these drawbacks, we propose the Multi-level Heterogeneous unsupervised DA method, termed MHDA, which unifies the instance-level DA based on cycle consistency, the feature-level DA based on contrastive learning, and the decision-level DA based on task consistency into a framework to more effectively handle the complex domain shift and heterogeneous data. After that, extensive DA experiments are conducted on the ISPRS dataset, the BigCity dataset constructed by ourselves, and the WHU dataset, in order to explore the effect of each module in MHDA, the necessity of heterogeneous DA, and the effectiveness of multi-level DA. And the results demonstrate that MHDA can achieve superior performance on the remote sensing image segmentation task, compared with several state-of-the-art DA methods.