The risk of arsenic (As) contamination from gold mining is a long-term environmental concern for mines worldwide. Researchers have mainly focused on As contamination induced by tailings, however, less attention has been paid mineralogically to differentiate the fate of As among different As-bearing matrices. This paper presents a detailed study of the mineralogical and morphological features of three typical As-bearing matrices (waste rock, ores, and tailings) using bulk chemical, microscopic and spectroscopic analyses, and reveals the geochemical behavior of As in those matrices. Results from mineral composition identified by RoqSCAN revealed that the matrices were dominated by quartz, k-feldspar, albite, muscovite, and clay minerals, with subordinate ankerite, chlorite, smectite, hematite, arsenopyrite, pyrrhotite, apatite, pyrite, halite, and calcite. The sequential extraction scheme indicated that As in waste rock, ores and tailings was mainly hosted in arsenopyrite. Microscopic analysis observed that waste rock was significantly different from the ores and tailings in terms of mineralogical and morphological characteristics. For waste rock, from arsenopyrite to hematite, As content decreased from 46.12 wt% to 3.54 wt%. However, arsenopyrite presented as unweathered euhedral crystals or slight fragmentation in ores and tailings and a narrower oxidation rims than that of waste rock. The leaching test of SPLP showed that the highest As leaching was found in waste rock (0.246 mg/L) which was significantly higher than those in ores (0.080 mg/L) and tailings (0.148 mg/L). The As in waste rock displayed weaker geochemical stability than in ores and tailings, as supported by mineralogy analysis. Health risk assessment suggested waste rock had a higher health risk for both adults and children compared with ores and tailings. These findings reaffirm that understanding of As fate among different source materials is paramount for securing humans from As hazards. More must be done to decelerate the continuous oxidation of waste rock, thus mitigating As release into nature.