Multi-object tracking (MOT) has received constant attention from researchers with the development of deep learning and person re-identification (ReID). However, the occlusion caused tracking failure is still far from solved. In this paper, we propose a Hard-Soft Attention Network (HSAN) to improve the ReID performance and get robust appearance features of different targets. The pose information and attention mechanism are combined to distinguish between challenging targets. Besides, the unary and binary costs are constructed to ensure consistency and long-term tracking, which consider not only the appearance motion affinity of single tracks, but also the interactions between neighboring tracks. For that we cluster the tracks into different groups and choose reliable tracks as anchors to establish the two types of costs. Our HSAN appearance model is evaluated on the Market-1501, DUKE and CUHK03 ReID datasets and the MOT tracking method is conducted on MOTChallenge 15, 16 and 17. The experimental results demonstrate that our method can improve tracking accuracy and reduce fragments. (c) 2021 Elsevier B.V. All rights reserved.