Facial Action Unit (AU) recognition is an essential step in the facial analysis. A facial image has one or more AU(s). Given an AU, the number of images without the AU is far greater than that of images with the AU. So, AU recognition is not only a sample imbalance problem but also a multi-label learning problem. For the two problems, we proposed a novel Multi-label Slope Rate (MSR) loss function and an Advanced-MSR (Ad-MSR) loss function in deep network architecture to recognize AU. For other characters of AU recognition, a local convolution and residual units are used in the architecture. The experimental results on two expression databases labeled AU show that the proposed loss functions not only address overfitting of the network on the training set and enhancing the generalization ability on the test set. The proposed architecture also gets well performance in the databases. (C) 2019 Elsevier B.V. All rights reserved.