Simple Summary Microvascular invasion (MVI) is an independent risk factor for postoperative recurrence of hepatocellular carcinoma (HCC). Preoperative knowledge of MVI would assist with tailored surgical strategy making to prolong patient survival. Previous radiological studies proved the role of noninvasive medical imaging in MVI prediction. However, hitherto, deep learning methods remained unexplored for this clinical task. As an end-to-end self-learning strategy, deep learning may not only achieve improved prediction accuracy, but may also visualize high-risk areas of invasion by generating attention maps. In this multicenter study, we developed deep learning models to perform MVI preoperative assessments using two imaging modalities-computed tomography (CT) and gadoxetic acid-enhanced magnetic resonance imaging (EOB-MRI). A head-to-head prospective validation was conducted to verify the validity of deep learning models and achieve a comparison between CT and EOB-MRI for MVI assessment. The findings put forward a better understanding of MVI preoperative prediction in HCC management. Microvascular invasion (MVI) is a critical risk factor for postoperative recurrence of hepatocellular carcinoma (HCC). Preknowledge of MVI would assist tailored surgery planning in HCC management. In this multicenter study, we aimed to explore the validity of deep learning (DL) in MVI prediction using two imaging modalities-contrast-enhanced computed tomography (CE-CT) and gadoxetic acid-enhanced magnetic resonance imaging (EOB-MRI). A total of 750 HCCs were enrolled from five Chinese tertiary hospitals. Retrospective CE-CT (n = 306, collected between March, 2013 and July, 2019) and EOB-MRI (n = 329, collected between March, 2012 and March, 2019) data were used to train two DL models, respectively. Prospective external validation (n = 115, collected between July, 2015 and February, 2018) was performed to assess the developed models. Furthermore, DL-based attention maps were utilized to visualize high-risk MVI regions. Our findings revealed that the EOB-MRI-based DL model achieved superior prediction outcome to the CE-CT-based DL model (area under receiver operating characteristics curve (AUC): 0.812 vs. 0.736, p = 0.038; sensitivity: 70.4% vs. 57.4%, p = 0.015; specificity: 80.3% vs. 86.9%, p = 0.052). DL attention maps could visualize peritumoral high-risk areas with genuine histopathologic confirmation. Both DL models could stratify high and low-risk groups regarding progression free survival and overall survival (p < 0.05). Thus, DL can be an efficient tool for MVI prediction, and EOB-MRI was proven to be the modality with advantage for MVI assessment than CE-CT.