The phase retrieval method based on random phase modulation can wipe out any ambiguity and stagnation problem in reconstruction. However, the two existing reconstruction algorithms for the random phase modulation method are suffering from problems. The serial algorithm from the spread-spectrum phase retrieval method can realize rapid convergence but has poor noise immunity. Although there is a parallel framework that can suppress noise, the convergence speed is slow. Here, we propose a random phase modulation phase retrieval method based on a serial-parallel cascaded reconstruction framework to simultaneously achieve quality imaging and rapid convergence. The proposed serial-parallel cascaded method uses the phased result from the serial algorithm to serve as the initialization of the subsequent parallel process. Simulations and experiments demonstrate that the superiorities of both serial and parallel algorithms are fetched by the proposed serial-parallel cascaded method. In the end, we analyze the effect of iteration numbers from the serial process on the reconstruction performance to find the optimal allocation scope of iteration numbers.