Unsupervised domain adaptation (UDA) for the semantic segmentation of remote sensing images is challenging since the same class of objects may have different spectra while the different class of objects may have the same spectrum. To address this issue, we propose a class-aware generative adversarial network (CaGAN) for UDA semantic segmentation of multisource remote sensing images, which explicitly models the discrepancies of intraclass and the interclass between the source domain images with labels and the target domain images without labels. Specifically, first, to enhance the global domain alignment (GDA), we propose a transferable attention alignment (TAA) procedure to add more fine-grained features into the adversarial learning framework. Then, we propose a novel class-aware domain alignment (CDA) approach in semantic segmentation. CDA mainly includes two parts: the first one is adaptive category selection, which is to alleviate the class imbalance and select the reliable per-category centers in the source and target domains; the second one is adaptive category alignment, which is to model the intraclass compactness and interclass separability from source-only, target-only, and joint source and target images. Finally, the CDA plays as a penalty of GDA to train GaGAN in an alternating and iterative manner. Experiments on domain adaptation of space to space, spectrum to spectrum, both space-to-space and spectrum-to-spectrum data sets demonstrate that CaGAN outperforms the current state-of-the-art methods, which may serve as a starting point and baseline for the comprehensive applications of semantic segmentation in cross-space and cross-spectrum remote sensing images.