Video summarization targets the challenge of finding the small est subset of frames, while still conveying the whole story of a given video. Thus it is of great significance for large-scale video understanding, allowing efficient processing of the large amount of videos that are uploaded every day. In this paper, we introduce a Dilated Temporal Relational Adversarial Network (DTR-GAN) to achieve frame-level video summarization. The dilated temporal relational units in the generator aim to exploit multi-scale temporal context in order to select key frames. To ensure that the model pre dicts high quality summaries, we present a discriminator that learns to enhance both the information completeness and compactness via a three-player loss. Experiments on the public TVSum dataset demonstrate the effectiveness of the proposed approach.