Estimating the 6D pose of a known object has very important applications in augmented reality and robot operations. This problem is challenging because of the clutter of the scene, the diversity of objects, and the complexity of lighting and textures. In this work, we propose a deep learning architecture for 6D object pose estimation, and a neural network that can predict object movement. By learning to predict the relative pose between the observation of current frame and the rendered image of previous prediction, the pose of the object can be tracked robustly for a long time. We have also introduced an efficient way of representing object motion, which can reduce the influence of the field of view and object scale so that this method has a strong cross-dataset generalization. We have conducted a lot of experiments on the LINEMOD dataset, the OccludedLINEMOD dataset, and the YCB dataset to show that this method can provide accurate pose estimation using only color images as input while being highly robust to occlusion.