Discrimination of surface textures using tactile sensors has attracted increasing attention. Intelligent robotics with the ability to recognize and discriminate the surface textures of grasped objects are crucial. In this paper, a novel method for surface texture classification based on tactile signals is proposed. For the proposed method, first, the tactile signals of each channel (X, Y, Z, and S) are decomposed based on empirical mode decomposition (EMD). Then, the intrinsic mode functions (IMFs) are obtained. Second, based on the multiple IMFs, the sample entropy is calculated for each IMF. Therefore, the multi-IMF sample entropy (MISE) features are obtained. Last but not least, based on the two public datasets, a variety of machine learning algorithms are used to recognize different textures. The results show that the SVM classification method, with the proposed MISE features, achieves the highest classification accuracy. Undeniably, the MISE features with the SVM method, proposed in this paper, provide a novel idea for the recognition of surface texture based on tactile perception.