Orthographic processing is a critical stage in visual word recognition. However, the white-matter pathways that support this processing are unclear, as prior findings might have been confounded by impure behavioral measures, potential structural reorganization of the brain, and limited sample sizes. To address this issue, we investigated the correlations between the integrity of 20 major tracts in the whole brain and the pure orthographic index across 67 patients with short-term brain damage. The integrity of the tracts was measured by the lesion volume percentage and the mean fractional anisotropy value. The orthographic index was calculated as the residual of the orthographic tasks after regressing out corresponding nonorthographic tasks and the orthographic factor from the principal component analysis (PCA) on the basis of four orthographic tasks. We found significant correlations associated with the left inferior longitudinal fasciculus (ILF), even after controlling for the influence of potential confounding variables. These observations strengthen evidence for the vital role of the left ILF in orthographic processing.