3D object detection plays a pivotal role in driver assistance systems and has practical requirements for small storage and fast inference. Monocular 3D detection alternatives abandon the complexity of LiDAR setup and pursues the effectiveness and efficiency of the vision scheme. In this work, we propose a set of anchor-free monocular 3D detectors called MonoPoly based on the keypoint paradigm. Specifically, we design a polynomial feature aggregation sampling module to extract multi-scale context features for auxiliary training and alleviate classification and localization misalignment through an attention-aware loss. Extensive experiments show that the proposed MonoPoly series achieves an excellent trade-off between performance and model size while maintaining real-time efficiency on KITTI and nuScenes datasets.