The calibration of the handheld target probe tip center position (PTCP) is an essential procedure in a target-based vision measurement system (T-VMS). At present, handheld PTCP calibration methods typically work via the least squares (LS) method, which is easily affected by noise and may generate a sizable error if placed improperly. This paper proposes a regularized total least squares (RTLS) method for handheld target PTCP calibration. Feature points on the handheld target are first subjected to a robust matching strategy, and then positioned precisely by center extraction deviation compensation. Fixed-radius constraint equations derived from the three-dimensional coordinates of feature points arc then used to establish an errors in variables (EIV) model of the PTCP. Finally, Tikhonov-regularized and L-curve methods are applied to search for the optimal solution of the EIV model, i.e. the PTCP coordinates. The proposed method was applied in the laboratory and on site to test its accuracy. Practical data demonstrated that it can calibrate the PTCP of a handheld target effectively and with better accuracy.