Improving the photoelectric conversion efficiency of organic solar cells (OSCs) is an essential issue for large scale commercial applications. Introducing the fluorine (F) atoms onto the electron-accepting unit and/or electron-donating unit on donor (D)-acceptor (A) type conjugated polymer is an effective strategy to enhance the overall solar cell performance. Here, the optimum sites for fluorination in favor of boosting the optical and photovoltaic properties of polymers are fully investigated by density functional theory (DFT) calculation and molecular dynamics (MD) simulation. Especially, the interchain charge mobility is calculated by means of Marcus theory, to investigate the charge transport process quantitatively. The results of theoretical modeling indicate that, except for the enhanced absorption spectra and deep lying HOMO level, more importantly, a higher hole mobility could be achieved when F is introduced onto the 3,6 position (outer site) of quaterthiophene for two reasons: (1) more effective face-on orientation formed by subtle manipulation of interchain pi-pi stacking pattern; (2) reduction of reorganization energy during the process of charge transport. These results elucidate that specific fluorination sites can influence the photoelectric properties of donor polymers, highlight the fluorination effect on the interchain interactions and orientation correlations in governing the key parameters of OSCs, and raise the hope of achieving even higher efficiencies by means of rational molecular designing.