Accurate three-dimensional measurement of potholes is a significant concern in road maintenance. However, the assessment of road potholes still relies heavily on human inspectors to make maintenance suggestions. To realize full-field measurement of pavement pothole automatically, a high-precision volume measurement method based on second-order Taylor expansion is proposed, where the second-order Taylor estimation of each point is converted into a convolution operation. On the one hand, this method discards the second-order fitting of the surface, which greatly reduces the computational complexity. On the contrary, the second-order Taylor estimation is not restricted by the surface shape because it only depends on the distribution of the adjacent points. Experiments on synthesized and real point sets demonstrate that the proposed method outperforms the state-of-the-art methods under various point cloud shapes.