Face recognition has made great progress with the development of deep learning. However, video face recognition (VFR) is still an ongoing task due to various illumination, low-resolution, pose variations and motion blur. In this paper, we propose a novel Attention-Set based Metric

Learning (ASML) method for VFR. It is a promising and generalized extension of Maximum Mean Discrepancy with Memory Attention Weighting inspired by Neural Turing Machine. ASML can be naturally integrated into Convolutional Neural Networks, resulting in an end-to-end learning scheme. Our method achieves state-of-the-art performance for the task of video face recognition on three widely used benchmarks including YouTubeFace, YouTube Celebrities and Celebrity-1000.