In this paper, we present a realtime natural interaction system by using Kinect sensor. It can stability and smoothly control hand like mouse by user's holding hands, and implement common mouse operations such as 'clicking', 'dragging' and 'dropping' so on. Our interaction system is made of several novel technique. It can identify the user's interaction with intent by detecting the engaged/disengaged gestures, and distinguish the primary user in the crowd, build a physical interaction zone to map the user's 3D hand position to 2D screen position in a timely and user-friendly manner. Our main contribution is to combine gesture recognition and gesture-tracking, and to implement an interaction application system with the details.