Magnetic particle imaging (MPI) realizes functional imaging by generating nonlinear response signals in the magnetic field through magnetic nanoparticles. MPI is a highly sensitive tracer-based imaging technology, which opens new possibilities for promising biomedical applications. However, in practice, the background signals, including random thermal noise from MPI receive chains and the harmonic interference caused by nonlinear components under a high-frequency excitation field, restrict the MPI performance. In this study, we propose a learning-based method by training a multi-scale dual-domain network to effectively filter the MPI signals. In the model, a multi-channel filtering module was designed to suppress the noise-related features in time and frequency domains. We constructed four different MPI signal datasets including simulated and measured noises acquired from a homemade MPI system to verify our method. The signal filtering tests were performed on synthetic and measured data. The experimental results indicated that our method can achieve the best performance among state-of-art signal filtering methods. Especially, on the dataset containing measured noise, our proposal improved the signal-to-noise ratio from 6.88 dB to 29.11 dB. Moreover, the percentage root means square difference was reduced from 51.26 to 3.96 and the root mean square error was reduced from 65.07 × 1e-4 to 5.29× 1e-4.