Boundary separation of operational regions would be helpful for unmanned surface vehicles deployed in dynamic outdoor environments. However, the feasibility and accuracy of current obstacle avoidance methods based on conventional optical images are comparatively poor for unmanned surface vehicle applications, with complicated natural illumination as one of the main sources of error. In this article, a new optical waterline detection method is proposed by combining shadow verification and global optimization (energy minimization). The method is then validated using an actual unmanned surface vehicle operating in outdoor environments. First, the basic principles of intrinsic image are introduced and then employed to evaluate the threshold for background segmentation so that the influence of complicated intensity distribution on the original image is reduced. The properties of different types of shadows are compared, and the basic principles of shadow verification are used to classify the different object regions. Subsequently, the intensity contrast between the shadow and non-shadow regions is used to measure the waterline position based on the relationship between the illumination and the shadow formation. Furthermore, the waterline detection problem is transformed into a problem involving the optimization of energy (minimization) described using differential equations. Finally, experiments are conducted with a series of practical images captured by the unmanned surface vehicle. The experimental results demonstrate the feasibility and robustness of the proposed method.