Accurate waterline separation of the sailing area is the base of unmanned surface vehicle (USV) systems moving in a complicated environment. However, shadows resulted from different natural illumination is one of the main error sources. This paper proposes a waterline separation method based on shadow processing with respect to optical images and presents a practical application to a USV system for validation. First, the basic principles of an intrinsic image is used to reduce the intensity influence of different shadows. Then, segmentation and shadow verification based on shadow analysis were introduced to separate and classify shadow regions. Finally, the problem of waterline separation is transformed into an optimization problem of mathematical energy minimum described by differential equations. In the practical application, the waterlines of many images captured by an USV system moving along an inland river are separated with the proposed method, and the experimental results showed that the proposed algorithm is effective and robust.