This paper tries to solve anomaly detection, a very important issue in ensuring the safe and stable operation of power system. As the proportion of abnormal data in the operation of power system is very small, a one-class support vector machine (OCSVM) is adopted in this paper, which is suitable for classification of unbalanced data. However, the performance of OCSVM is sensitive to its parameters, and an unsuitable choice will decrease the classification accuracy and generalization ability of it. In this paper, particle swarm optimization (PSO) is used to optimize the parameters of OCSVM. The original PSO algorithm converges slowly and easily falls into local optimum. To overcome this issue, this paper proposes an improved PSO algorithm for parameters optimization, in which adaptive speed weighting and adaptive population splitting are introduced to improve the convergence speed of the algorithm and help the algorithm jump out of the local optimal position. Experiments on standard benchmarks and real power system experimental data sets demonstrate the effectiveness of the proposed algorithm.