Convolutional neural network is widely used in image fusion. However, the deep learning framework is only applied in some part of the fusion process in most existing methods. To generate a full end-to-end image fusion pipeline, a Y-shaped Generator model based on Generative Adversarial Network for infrared and visible image fusion is proposed. The idea of this method is to establish an adversarial game between the generator and the discriminator. The generator consisting of two Pyramid networks and three convolutional layers works as an autoencoder to improve the characteristic information of the fused images. As for the discriminator, it adopts a network structure similar to the Visual Geometry Group (VGG) network. The loss function uses the ratio loss to control the trade-off among generation loss and reconstruction loss. Results on publicly available datasets demonstrate that our method can improve the quality of detail information and sharpen the edge of infrared targets. © 2021 SPIE