Early fire and smoke detection with computer vision have attracted much attention in recent years, and a lot of fire detectors based on deep neural network have been proposed to improve the detection accuracy. However, most current fire detectors still suffer from low detection accuracy caused by the multi-scale variation of the fire and smoke, or the high false accept rate due to the fire-like or smoke-like objects within the background. In this paper, to address the above challenges, we propose an effective real-time fire detection network (AERNet) with two key functional modules, which achieves a good tradeoff between the detection accuracy and speed. First, we employ a lightweight backbone network Squeeze and Excitation-GhostNet (SE-GhostNet) to extract features, which can make it easier to distinguish the fire and smoke from the background and reduce the model parameters greatly. Second, a Multi-Scale Detection module is constructed to selectively emphasize the contribution of different features by channel and space. Finally, we adopt the decoupled head to predict the classes and locations of fire or smoke respectively. In the experiment, we propose a more challenging dataset Smoke and Fire-dataset (SF-dataset) to evaluate the proposed algorithm, which includes 18,217 images. And the results show that the proposed method outperforms most SOTA methods in detection accuracy, model size, and detection speed.