Terahertz Time-Domain Spectroscopy (THz-TDS) has been a powerful tool for material characterization. The Fabry-Perot (FP) effect caused by the multiple reflections within the sample exists inherently, which makes the echoes contaminated. In the case of thin samples, different reflection signals may merge together and even the echoes corresponding to the main reflection disappears, making the characterization improper or impossible. To deal with this problem, a novel time-domain optimization method is proposed for removing the FP effect. By introducing a new term which describes the variation of the waveform in time-domain, not only the complex refractive index but also the thickness of material are obtained for a very thin silicon wafer (~100 m) whose thickness is less than 2/3 lambda at the central frequency of the THz-TDS system. The accuracy of the algorithm is quite comparable to the iterative method, which is considered to be the most accurate one at the moment, but it is more than 100 times' faster, with a better convergence performance. Hence, it is promising for rapid material scenarios, like the in-line quality control and imaging-related applications.