Deep neural networks has been widely used in industrial equipment fault diagnosis. The accuracy of deep neural network is usually proportional to the complexity, but the high inference delay and energy consumption caused by the complex model make it difficult to be applied in the industrial environment of real-time demand. At the same time, in the diagnosis of industrial equipment, different categories of samples have unbalanced characteristics in terms of number, difficulty of identification, and demand of identification. In order to solve this problem, this paper designs Multi-Branch Neural Network (MBNN), which is a new type neural network architecture that can use the unbalance of sample categories in industrial equipment fault diagnosis for fast inference. MBNN has multiple sub-networks with different complexity, and each branch is responsible for processing different categories of samples. Categories with large numbers, easy to process, and high demand of identification are processed through simple branches, such as normal samples. Categories with small numbers, difficult to identification, and low demand of identification are processed through complex branches, such as potential failure samples. The feasibility of MBNN has been verified on motor bearing fault diagnosis and gearbox fault diagnosis, and its performance has been evaluated on multiple computing platforms. The results show that MBNN can greatly improve the inference speed while ensuring the recognition accuracy, especially on resource-constrained platforms.