Pavement defect detection is of profound significance regarding road safety, so it has been a trending research topic. In the past years, deep learning based methods have turned into a key technology, with advantages of high accuracy, strong robustness, and adaptability to complex pavement environments. This paper first reviews the methods based on image processing and 3D imaging. As for image-based processing techniques, they can be classified into three types based on how to label the collected data: fully supervised learning, unsupervised learning, and other methods. Different methods are further classified and compared with each other. Second, the pavement detection methods based on 3D data are sorted out, thereby summarizing their benefits, drawbacks, and application scenarios. Third, the study proposed the major challenges in the field of pavement defect detection, introduced validated datasets and evaluation metrics. Finally, on the basis of reviewing the literature in pavement defect detection, the promising direction is put forward.