Human action recognition in video is an active research topic in computer vision. However, with the growing convenience of capturing and sharing videos, there are a growing variety of human action datasets with substantial amount of videos make human action recognition challenging problems, which can be solved by key frame extraction. Feature Clustering methods are usually employed to extract key frames. One difficulty is caused by the large variety of visual content in videos, makes hand-craft feature is not always effective, since there are no fixed descriptors can describe all video cases. Another difficulty is that traditional clustering algorithms are sensitive to the choice of initial clustering centers. An Unsupervised feature learning and clustering method is proposed for key frame extraction on human action recognition, Stacked auto-encoder(SAE) is trained using videos from 10 different human actions, after training, SAE is used as a feature extractor to learn features representing human actions. Affinity Propagation Clustering algorithm is used to select key frames from video sequences. Experiments using a variety of videos demonstrate that our method can be effectively summarizing video shots considering different human actions.