Age estimation of unknown persons is a challenging pattern analysis task due to the lacking of training data and various aging mechanisms for different individuals. Label distribution learning-based methods usually make distribution assumptions to simplify age estimation. However, since humans with different genders, races and/or any other situations may influence their facial aging appearances, age label distributions are often complicated and difficult to be modeled in a parameter way. In this paper, we propose a Label Refinery Network (LRN) with two concurrent refinery processes: label distribution refinery and slack regression refinery. Label refinery network aims to learn age label distributions progressively in an iterative manner. In this way, we can adaptively obtain the specific age label distributions for different facial images without making strong assumptions of fixed distribution formulations. To further utilize the correlations among age labels, we accordingly propose a slack regression refinery to convert the age label regression into the age interval regression. Extensive experiments on three popular datasets, including Morph, ChaLearn15 and MegaAge-Asian demonstrate the superiority of our method.