Nigella glandulifera Freyn belongs to the Ranunculaceae family of plants. It is an annual herbaceous plant widely distributed in the southwest and western regions of China. The seeds and oil of N. glandulifera have been used as a medicine to treat a variety of diseases. Unfortunately, the extraction of bioactive components and their inherent phytochemical characterization is rarely investigated using optimum condition of emerging extraction. In the study, we compared the microwave-assisted method to traditional extraction methods, which has the shortest time, highest yield and greatest biological activity. We performed single factor experiments with comprehensive assessments and used Box-Behnken design with a response surface methodology to assess the optimal conditions. The optimal extraction method was as follows; solid-to-liquid ratio of 1:20 g/mL, ethanol concentration of 70%, extraction time of 35.00 min, and extraction temperature of 66 degrees C. Using these conditions, we further analyzed the spectrum-effect relationship using gray correlation and chemical composition using ultra-high-performance liquid chromatography coupled with high resolution hybrid quadruple-orbitrap mass spectrometry (UHPLC-Q-Orbitrap-HRMS). Therefore, the gray relationship of 17 compounds was analyzed greater than 0.95 which may have synergistic effects on the inhibiting of aldose reductase. In addition, thirty compounds were identified, including alkaloids, flavonoid glucosides, triterpenoid saponins and phenolic compounds, all of which may have a therapeutic effect on diabetic complications. The results of this study may offer a time-saving and convenient method to acquire bioactive components to improve and treat diabetic complications.