This paper presents a real-time vision based robot teleoperation system that consists of a three-dimensional (3D) vision subsystem and a slave robot which are connected by LAN. The vision subsystem utilizes an Asus Xtion Pro Live camera to get the 3D data of the operation scene. The vision system is used to determine the position and orientation of a four-ball feature frame held by the operator. Then the position and the orientation are used to control a remote robot. In the vision subsystem, support vector domain description (SVDD) is adopted to detect the balls on the feature frame. In this paper, we propose a novel colour table to speed up the detection procedure and utilize Kalman filter for ball tracking to reduce the detection area for further acceleration. The operator can see the motion of the robot which enables the operator make some corrections constantly.The SVDD colour classifier and the teleoperation system are tested in experiments.