Convenient and efficient human-computer interaction has always been the goal pursued by researchers. In this paper, we present a vision-based handwriting interaction device for non-touch screens called Visual Pencil. Unlike the conventional interactive method that captures the user’s gestures using a motionless visual sensor, Visual Pencil directly mounts the visual sensor in the device. More specifically, the visual data are processed by the feature detection algorithm, and the movement of the pen tip in a 2D plane is calculated, so as to control the cursor and achieve the purpose of interaction. We further define performance indicators for evaluating the design and select more reasonable parameters through specific experiments to achieve the best interaction performance. The experiments verify that the proposed design is able to fulfill the desired operations. The significance of this design lies in that it integrates the functions of the mouse and smart pen, thus removing the requirement of more auxiliary equipment. This work sheds light on the interaction with non-touch screen computers.