Magnetic particle imaging (MPI) is an emerging medical imaging technique that has high sensitivity, contrast and excellent depth penetration. In x-space MPI reconstruction, the reconstructed native image can be modeled as a convolution of the magnetic particle concentration with a point-spread function (PSF). The deconvolution is practical and valuable as a post-processing way to deblur the native image. However, to accurately measure or model the PSF used for deconvolution is challenging due to the imperfection of hardware and magnetic particle relaxation. The inaccurate PSF may lead to the loss of the content structure of the MPI image. In this study, we developed a dual adversarial framework with contrastive constraint (DC_GAN) to deblur the MPI image. We evaluate the performance of the proposed DC_GAN model on simulated and real data. Experimental results confirm that our model performs favorably against the deconvolution method that are mainly used for deblurring the MPI image.