Surface-enhanced Raman Spectroscopy (SERS) is extensively implemented in drug detection due to its sensitivity and non-destructive nature. Deep learning methods, which are represented by convolutional neural network (CNN), have been widely applied in identifying the spectra from SERS for powerful learning ability. However, the local receptive field of CNN limits the feature extraction of sequential spectra for suppressing the analysis results. In this study, a hybrid Transformer network, TMNet, was developed to identify SERS spectra by integrating the Transformer encoder and the multi-layer perceptron. The Transformer encoder can obtain precise feature representations of sequential spectra with the aid of self-attention, and the multi-layer perceptron efficiently transforms the representations to the final identification results. TMNet performed excellently, with identification accuracies of 99.07% for the spectra of hair containing drugs and 97.12% for those of urine containing drugs. For the spectra with additive white Gaussian, baseline background, and mixed noises, TMNet still exhibited the best performance among all the methods. Overall, the proposed method can accurately identify SERS spectra with outstanding noise resistance and excellent generalization and holds great potential for the analysis of other spectroscopy data.