Seepage-induced fines migration under rainfall infiltration is a main cause leading to shallow failures in loose colluvial slopes. To describe the full process of fines migration within unsaturated soils during rainfall infiltration and the associated hydro-mechanical behaviors, a seepage-erosion-deformation coupled formulation is proposed in this paper. The governing equations proposed are implemented into a finite element code and used to investigate the influences of skeleton deformation on the rainfall infiltration process through unsaturated soil columns. The numerical results were presented in detail for a better understanding of the rainfall-induced fines migration process within unsaturated soils. Further, the obtained results are integrated into an infinite slope model for slope stability analysis. The results show that, the skeleton deformation will affect the rainfall infiltration rate and hence the timing of slope failures; meanwhile their influences are more evident if the fines deposition process is taken into account. Moreover, the slope stability could be reduced gradually due to the soil strength loss along with loss of fine particles. Therefore, particular attentions should be paid to analyzing the stability of soil slopes susceptible to internal erosion.