For endovascular aneurysm repair (EVAR), real-time and accurate segmentation and tracking of interventional instruments can aid in reducing radiation exposure, contrast agents, and procedure time. Nevertheless, this task often comes with the challenges of the slender deformable structures with low contrast in noisy X-ray fluoroscopy. In this paper, a novel efficient network architecture, termed FRR-Net, is proposed for real-time catheter segmentation and tracking. The novelties of FRR-Net lie in the manner in which recurrent convolutional layers ensure better feature representation and the pre-trained lightweight components can improve model processing speed while ensuring performance. Quantitative and qualitative evaluation of images from 175 X-ray sequences of 30 patients demonstrate that the proposed approach significantly outperforms simpler baselines as well as the best previously-published result for this task, achieving the state-of-the-art performance