Magnetic particle imaging (MPI) is a novel medical imaging technology, which uses the nonlinear magnetization response of super-paramagnetic iron oxide (SPIOs) to describe the distribution of SPIOs in the target region. X-space method is a widely used image reconstruction method of MPI, which directly reconstruct in spatial domain. However, due to its inherent limitation of disregarding the physical properties of real systems, native X-space reconstructed MPI images are usually very blurry which limits the subsequent clinical use. In this study, we propose a novel multi-scale adaptive transformer (MSA-transformer) for image deblurring and recover the detail information. We perform experiments on both simulated datasets and real data to evaluate of our method. The experimental result shows, our method outperforms existing methods including methods used in MPI and some prevalent restoration methods used in computer vision.