Open-vocabulary object detection aims to detect unseen objects beyond training set, and the recent two-stage detectors that distill knowledge from zero-shot image classification models, ie, CLIP, have achieved state-of-the-art performance in this area. However, when it comes to more efficient one-stage detectors, the missing of class-agnostic object proposals in distillation leads to an obvious performance gap between these two kinds of detectors. In this paper, a hierarchical knowledge distillation mechanism, namely HierKD, is explored to devise a fast and top-performing open-vocabulary one-stage detector. Besides the commonly used instance-level visual-to-visual knowledge distillation, it is also equipped with another weakly supervised global-level visual-to-language knowledge distillation with captions to learn novel category knowledge beyond training labels. The evaluation results on COCO dataset shows the proposed method significantly surpasses the previous best one-stage method with 70% and 33% AP50 gains under the Zero-shot Detection and Generalized Zero-shot Detection settings respectively. The code will be publicly available upon publication.