In wave-front coded imaging system, the phase mask placed in the pupil plane of the imaging system aims to reshape the PSF (point spread function) or OTF (optical transfer function) to realize DOF (depth of field) extension. How to design a suitable phase mask to provide a highly controlled response of system PSF or OTF is crucial to computational imaging application. Traditionally, AF (ambiguity function) is a powerful tool to assess the DOF extension effect generated by phase masks with known phase function. However, in this paper, we investigate an iterative optimization based procedure to recover the unknown phase mask using AF in a backward way. First, a set of desired PSFs or OTFs at different defocus planes is combined together to construct an initial estimate of AF. Second, the corresponding mutual function is calculated through Fourier transform. Third, SVD (singular value decomposition) is applied to the mutual function. Fourth, only the term corresponding to the biggest eigenvalue is kept and inverse Fourier transform is used to generate a new estimate of AF. Fifth, the input desired OTFs are used to update the newly estimated AF. This procedure iterates until the OTFs extracted from the estimated AF are highly consistent with the input ones using the MSE (meansquare-error) as criterion. In the paper, we systematically study this powerful procedure using numerical simulation and investigate the probability of recovering the rectangular non-separable phase masks. After that, experiments are carried out to justify the effectiveness of the procedure. © 2018 SPIE.