Endowing robots with tool manipulation skills helps them accomplish challenging tasks. While robots manipulate tools to achieve goals, the alignment of tools and targets is a noise-sensitive and contact-rich task. However, it is difficult to access the accurate pose of the tool and the target. When there is unknown noise in the observations, reinforcement learning can't be sure to perform well. In this paper, we define the easier-to-obtain accurate task-related information as anchor information and introduce a tool manipulation method based on reinforcement learning and anchor information, which can perform well when the observations include unknown noise. To evaluate the method, we build a simulated environment ToolGym, which includes four different kinds of tools and different noise sampling functions for each tool. Finally, we compare our method with baseline methods to show the effectiveness of the proposed method.