Direct-imaging spectra hold rich information about a planet's atmosphere and surface, and several space-based missions aiming at such observations will become a reality in the near future. Previous spectral retrieval works have resulted in key atmospheric constraints under the assumption of a gray surface, but the effect of wavelength-dependent surface albedo on retrieval has not been shown. We explore the influence of the coupling effect of cloud and wavelength-dependent surface albedo on retrieval performance via modeling suites of Earth-like atmospheres with varying cloud and surface albedo parameterizations. Under the assumption of known cloud scattering properties, the surface spectral albedos can be reasonably recovered when the surface cover represents that of Earth-like vegetation or ocean, which may aid in characterizing the planet's habitability. When the cloud scattering properties cannot be assumed, we show that the degeneracy between the cloud properties and wavelength-dependent surface albedo leads to biased results of atmospheric and cloud properties. The multiepoch visible-band observations offer limited improvement in disentangling this degeneracy. However, the constraints on atmospheric properties from the combination of the UV band (R ∼ 6) + visible band (R ∼ 140) are consistent with input values to within 1σ. If short-bandpass data are not available, an alternative solution to reduce the retrieval uncertainties would be to have the prior constraints on the planetary cloud fraction with less than 20% uncertainty.