Feature selection is an important procedure in machine learning because it can reduce the complexity of the final learning model and simplify the interpretation. In this paper, we propose a novel non-linear feature selection method that targets multi-class classification problems in the framework of support vector machines. The proposed method is achieved using a kernelized multi-class support vector machine with a fast version of recursive feature elimination. The proposed method selects features that work well for all classes, as the involved classifier simultaneously constructs multiple decision functions that separates each class from the others. We formulate the classifier as a large optimisation problem, and iteratively solve one decision function at a time, leading to a lower computational time complexity than when solving the large optimisation problem directly. The coefficients of the classifier are then used as a ranking criterion in the accelerated recursive feature elimination by adding batch elimination and a rechecking process. Experimental results on several datasets demonstrate the superior performance of the proposed feature selection method.