Single photon 3D imaging is an emerging topic for optronic sensing under extreme scenarios (e.g. spaceborne altimeter, long range imaging). However, such technique suffers from low photon counts and strong noise, which is due to either strong attenuation from the environment or reduced acquisition time. Although state-of-the-art algorithms have been proposed to achieve high resolution results from corrupted single photon 3D data, the trade-off between the restoration performance and computational complexity remains challenging. This paper presents a fast and robust restoration approach for single photon 3D data, which adaptively smooth the sparse and noisy histogram by applying a parameterized kernel and finally reconstruct the 3D image using matched filter. The implementation can be fast as the core step of the processing is generalized as a 3D convolution that can be solved by Fast Fourier Transform (FFT). The method is validated on various conditions and scenarios from Middbury dataset and real data, where the proposed method showed robust results as the competing state-of-the-art algorithms with fast implementation.