Contrast-enhanced ultrasound (CEUS) is an effective imaging tool to analyze spatial-temporal characteristics of lesions and diagnose or predict diseases. However, delineating lesions frame by frame is a time-consuming work, which brings challenges to analyzing CEUS videos with deep learning technology. In this paper, we proposed a novel U-net-like network with dual top-down branches and residual connections, named CEUSegNet. CEUSegNet takes US and CEUS part of a dual-amplitude CEUS image as inputs. Cross-modality Segmentation Attention (CSA) and Cross-modality Feature Fusion (CFF) are designed to fuse US and CEUS features on multiple scales. Through our method, lesion position can be determined exactly under the guidance of US and then the region of interest can be delineated in CEUS image. Results show CEUSegNet can achieve a comparable performance with clinicians on metastasis cervical lymph nodes and breast lesion dataset.