Underwater images often suffer from color distortion and resolution degradation due to the absorption and scattering of light. To deal with these problems, a novel underwater image restoration framework was designed to remove color deviation and background scattering successively. Firstly, an effective color correction method based on Lab color space is adopted to remove the color cast. Secondly, the color, saturation, and detail information are considered comprehensively to construct the scene depth map and edge map to estimate the transmission, and the relationship between the scattering coefficient and wavelength is used to modify the transmission map of each channel. In addition, a weighted least squares filter with adaptive smoothing parameters is introduced to estimate the local background light, which is applicable to the underwater scene with complex scattering. The experimental results show that the method performs better in color correction and detail enhancement, and the image quality is significantly improved.