The development of deep learning techniques has greatly benefited CNN-based object detectors, leading to unprecedented progress in recent years. However, the distribution variance between training and testing domains causes significant performance degradation. Labeling data for new scenarios is costly and time-consuming, so most existing domain adaptation methods perform feature alignment through adversarial training. While this can improve the accuracy of detectors in unlabeled target domains, the unconstrained domain alignment also negatively transfers the feature distribution, which compromises the recognition ability of the model. To address this problem, we propose the Knowledge Transfer Network (KTNet), which consists of object intrinsic knowledge mining and category relational knowledge constraint modules. Specifically, a binary classifier shared by the source and target domains is designed to extract common attribute knowledge of objects, which can align foreground and background features from different data domains adaptively. Then, we construct relational knowledge graphs to explicitly constrain the category correlations in the source, target, and cross-domain settings. These two modules guide the detector to learn object-related and domain-invariant representations, enabling the proposed KTNet to perform well in four commonly-used cross-domain scenarios. Furthermore, the ablation experiments show that our method is scalable to more complex backbone networks and different detection architectures.