Studies have shown that Tai Chi and conventional exercise can modify the brain through distinct mechanisms, resulting in different brain adaptations. Therefore, it is conceivable to speculate that these two exercise modalities may have different effects on improving cognitive function. This study was a parallel group, assessor-blinded, pilot randomized controlled trial comparing the effects of Tai Chi and conventional exercise on improving cognitive function in older persons with mild cognitive impairment (MCI). A total of 34 adults aged ≥ 50 years with MCI were randomized (1:1:1) to the Tai Chi group (TC, n = 10, 3 sessions of 60-min Yang-style Tai Chi training per week for 24 weeks), conventional exercise group (EX: n = 12, 3 sessions of 60-min fitness training per week for 24 weeks), or control group (CON: n = 12, no intervention). Global cognitive function assessed by the Hong Kong version of the Montreal Cognitive Assessment (MoCA-HK) and performance in various cognitive domains were examined at baseline, and 12 and 24 weeks of the intervention. Both exercise groups showed improved global cognitive function as measured by MoCA-HK compared with the control group after 12 and 24 weeks of the intervention, (all P < 0.001). Only TC achieved clinically relevant improvement on global cognitive function at week 12. Both exercise groups achieved clinically relevant improvements at the end of the interventions at week 24. Compared with EX, TC exhibited greater improvements on global cognitive function indicated by MoCA-HK after 12 weeks of the intervention (P < 0.001) and cognitive flexibility indicated by part B/A ratio score of the Trail Making Test throughout the study (all P < 0.05). Both interventions were equally effective in improving the other examined cognitive domains. Further studies are needed to substantiate the superior long-term benefits of Tai Chi on global cognitive function compared with conventional exercise, and dissect the underlying mechanisms of the two exercises on improving cognitive domains and the corresponding brain adaptations.