Multi-scale patterned assembly and integration is an important solution to break through the industrial application of carbon nanotubes (CNTs). The strategy based on self-assembly has yielded fruitful achievements in the practice of fabricating patterns such as two dimensional arrays. However, the mechanism of self-assembly of CNTs and the key factors that limit the fidelity of patterns have always been an unresolved issue. In this paper, all-atomic molecular models for studying the mechanism of hydrophilic/hydrophobic self-assembly are established and some key factors limiting the fidelity of SWNTs patterns are thoroughly investigated. The results show that chiral indices have little effect on the results of self-assembly. Secondly, edge effect determines the assembly sequence on hydrophilic surface and self-assembly will not occur when M-SWNTs are above the hydrophilic/hydrophobic interface. Thirdly, the assembly sequence of M-SWNTs in different layers of the solution is dominated by the competition of H2O molecules. At last, shielding effect is very important for the study of how to improve the density of SWNTs in the pattern. The explicitly results are not only helpful to understand many phenomena in self-assembly process at the atomic scale but also will provide meaningful guidance in fabrication of SWNTs patterns to prevent distortion.