In recent years, the Siamese network based trackers have achieved outperforming results in the metrics of speed and accuracy in terms of visual tracking. However, it is difficult for the Siamese networks to achieve the accurate location of rotating objects. This paper proposes an efficient network architecture named SiamPBN to resolve rotation problem in tracking. We find that the predicted center of the object deviates from the center of the predicted box while the object rotates continuously. Therefore, we add a centerness head to the network to improve the performance of handling rotating objects. In addition, SiamPBN obtains the predicted outputs by point-based instead of anchor-based method, thus avoiding the complex computation associated with anchors. Our proposed method has gained the excellent performance on the datasets, including GOT-10k, LaSOT and VOT2019. © 2021 IEEE.