Cerebral small vessel disease is the one of the most prevalent causes of vascular cognitive impairment. We aimed to find objective and process-based indicators related to memory function to assist in the detection of memory impairment in patients with cerebral small vessel disease. Thirty-nine cerebral small vessel disease patients and 22 healthy controls were invited to complete neurological examinations, neuropsychological assessments, and eye tracking tasks. Eye tracking indicators were recorded and analyzed in combination with imaging features. The cerebral small vessel disease patients scored lower on traditional memory task and performed worse on eye tracking memory task performance compared to the healthy controls. The cerebral small vessel disease patients exhibited longer visit duration and more visit count within areas of interest and targets and decreased percentage value of total visit duration on target images to total visit duration on areas of interest during decoding stage among all levels. Our results demonstrated the cerebral small vessel disease patients performed worse in memory scale and eye tracking memory task, potentially due to their heightened attentional allocation to nontarget images during the retrieval stage. The eye tracking memory task could provide process-based indicators to be a beneficial complement to memory assessment and new insights into mechanism of memory impairment in cerebral small vessel disease patients.