3D reconstruction is an important topic in computer vision with many applications, such as robotics and augmented reality. Since the raw depth images captured by consumer RGB-D cameras are often low-resolution, noisy, and incomplete. How to obtain high-quality 3D models with a consumer RGB-D camera is still a challenge for existing systems. In this paper, we propose a new depth super-resolution and completion (SRC) method implemented in deep learning framework, and build a high-quality 3D reconstruction system. We first improve the resolution of low-resolution (LR) depth image with a depth super-resolution network, and remove the outliers in high-resolution (HR) depth image based on gradient saliency. To further enhance the quality of HR depth image with the guide of HR color image, we learn surface normal and occlusion boundary images from the corresponding HR color image through two deep fully convolutional networks (FCN). In particular, the blurriness of HR color image is also detected and pixel-wise quantized. Finally, we obtain a completed HR depth image by optimizing the HR depth image with surface normal, occlusion boundary and color image blurriness. We have carried out qualitative and quantitative evaluations with baseline methods on public datasets. Experimental results demonstrate that our method has better performance both on single depth image enhancement and 3D reconstruction.