We observed an attribute entanglement phenomenon: samples with similar attributes but from different classes can easily result in recognition errors. This problem is an important cause that results in recognition errors. To address this problem, we propose a new loss function, namely the entanglement loss. It penalizes the compactness between the misclassified entangled samples and their misclassified class centers, such that the features of entangled samples are apart from the misclassified classes. The proposed loss function can effectively enhance the discriminative power of the deeply learned features, thus recognition performance can be significantly improved. Experimental results show that our method outperforms the previous state-of-the-art methods on PASCAL VOC 2012 Action and ASLAN datasets.