To meet the demand for rapid bacterial detection in clinical practice, this study proposed a joint determination model based on spectral database matching combined with a deep learning model for the determination of positive-negative bacterial infection in directly smeared urine samples. Based on a dataset of 8124 urine samples, a standard hyperspectral database of common bacteria and impurities was established. This database, combined with an automated single-target extraction, was used to perform spectral matching for single bacterial targets in directly smeared data. To address the multi-scale features and the need for the rapid analysis of directly smeared data, a multi-scale buffered convolutional neural network, MBNet, was introduced, which included three convolutional combination units and four buffer units to extract the spectral features of directly smeared data from different dimensions. The focus was on studying the differences in spectral features between positive and negative bacterial infection, as well as the temporal correlation between positive-negative determination and short-term cultivation. The experimental results demonstrate that the joint determination model achieved an accuracy of 97.29%, a Positive Predictive Value (PPV) of 97.17%, and a Negative Predictive Value (NPV) of 97.60% in the directly smeared urine dataset. This result outperformed the single MBNet model, indicating the effectiveness of the multi-scale buffered architecture for global and large-scale features of directly smeared data, as well as the high sensitivity of spectral database matching for single bacterial targets. The rapid determination solution of the whole process, which combines directly smeared sample preparation, joint determination model, and software analysis integration, can provide a preliminary report of bacterial infection within 10 min, and it is expected to become a powerful supplement to the existing technologies of rapid bacterial detection.