Deep Convolutional Neural Network (DCNN) has recently advanced state-of-the-art performance on vision-related tasks and its application is further extended to industrial fields. The paper focuses on the problem of fabric defect detection to which an efficient DCNN architecture is developed. In contrast to previous methods that directly apply existing DCNN models demonstrated on natural images to industrial images, the proposed Efficient Defect Detectors (EDDs) are sufficiently optimized with consideration of the characteristics of fabric surface images, i.e., resolution, defect appearance, etc. Firstly, lightweight backbone is suggested in EDD to improve computational efficiency without reduction in image resolution. Secondly, a new feature fusion strategy named L-shaped feature pyramid network (L-FPN) is proposed and utilized to make full use of low-level texture features which are demonstrated to be more important than high-level semantic features in defect recognition. Based on the configurations of lightweight backbone and L-FPN, we use only one hyper-parameter to jointly adjust the proportion of resources occupied by width, depth and input resolution so that a family of defect detectors under different resource constraints can be developed. Experiments are conducted on a large fabric dataset to demonstrated the effectiveness of EDDs. Compared with the recent state-of-the-art detector, EfficientDet-d3, EDD-d3 achieves higher mean Average Precision (mAP) (20.9 vs 19.9) but with fewer parameters. EDD-d3 has 8.59M parameters and 31.78B FLOPs (floating point operation per second), which respectively are 39.8% and 49.0% lower than EfficientDet-d3. The proposed EDDs achieve better tradeoff between accuracy and speed than previous methods. EDDs could be applied to fabric production sits with different resource restrictions, which demonstrates that EDDs have important application value.