Breast cancer diagnosis in multi-gigapixel whole slide images (WSIs) is an important task that highly relevant to cancer grading and prognosis. In recent years, many computer-aided diagnosis methods were proposed and achieved promising performance. However, they mostly suffer from heavy computational burden that becomes a significant barrier to clinical practice. Efficient solutions are urgently demanded but still less studied. In this paper, we propose a novel framework named EffiDiag for a fast and lightweight breast cancer diagnosis. To this end, a loss-modified U-net is developed at first to enable a fast suspected cancer Region Of Interest (ROI) localization. Therefore the subsequent patch-based classification, which commonly executes at the finest magnification hundreds of thousands times per WSI for cancer identification, could be carried out on these ROIs only rather than the whole WSI for speedup. Meanwhile, a super-efficient convolutional neural network (CNN) is devised to optimize the classification speed and resource consumption per classification. Experiments on the Camelyon16 benchmark demonstrate, by integrating the two contributions into a well-established approach, 47x inference acceleration is obtained with limited accuracy drop, yet with much less resource consumption even compared to popular lightweight networks.