Most of existing methods mainly focus on the food image recognition which assumes that one food image contains only one food item. However, in this paper, we present a system to detect a diversity of foods in refrigerator where multiple food items may exist. In view of the refrigerator environment, we propose a food detection framework based on the supervised transformer network. More specifically, the supervised transformer network, dotted as RectNet, is first proposed to automatically select the irregular food regions and transform them to the frontal views. Then, based on the rectified food images, we further propose an end-to-end detection network that predicts the categories and locations of food items. The proposed detection network, called Lite Fully Convolutional Network (LiteFCN), is evolved from the advanced object detection algorithm Faster R-CNN while several significant improvements are tailored to achieve a higher accuracy and keep inference time efficiency. To validate the effectiveness of each component of our method, we build a real-world refrigerator dataset with 80 classes. Extensive experiments demonstrate that our methods achieve the state-of-the-art results, which improves the baseline by a large margin, e.g., 3–5% in terms of F-measure. We also show that the proposed detection network achieve a competitive result on the public PASCAL VOC2007 dataset, which outperforms the Faster R-CNN by 2.3% with a higher speed.