This paper proposes a novel robust control scheme for tip trajectory tracking of a lightweight flexible single-link arm. The developed control scheme deals with the influence of tip payload changes and disturbances during the working process of the flexible arm, thus realizing the accurate tracking for the tip reference trajectory. The robust control scheme is composed of an inner loop and an outer loop. The inner loop adopts the traditional PD control, and an active disturbance rejection control (ADRC) with a sliding mode (SM) compensation is designed in the outer loop. Moreover, the sliding mode compensation is mainly used to cope with the disturbance estimation error from the extended state observer (ESO), by which the insensitivity to tip payload variations and strong disturbance resistance is achieved. Finally, some numerical simulations are performed to support the theoretical analysis. The results show that the system is more robust to the tip mass variations of the arm and more resistant to the external torque after adding the sliding mode robustness term to the ADRC.