This paper presents a new mechanism which is more effective for wearable devices to classify patient-specific electrocardiogram (ECG) heartbeats. In our method, a Generic Convolutional Neural Network (GCNN) is trained first using a large number of heartbeats without distinguishing patients. Based on the GCNN, fine-tuning technique is applied to modify the GCNN to a Tuned Dedicated CNN (TDCNN) for the corresponding individual. Notably, only the GCNN instead of common training data is required to be stored into wearable devices. Moreover, only fine-tuning with several seconds rather than dozens of minutes is needed before the TDCNN is used to monitor the long-term ECG signals in clinical. To accelerate the ECG classification, only the original ECG heartbeat is input to the CNN without other extended information from the neighbor heartbeats or FFT representation. A deeper CNN architecture with small-scale convolutional kernels is adopted to improve the speed and accuracy for classification. With deeper CNN, hierarchical features can be extracted to help improve the accuracy of ECG classification. The state-of-the-art performance on efficiency and accuracy for ECG classification over MIT-BIH dataset is achieved by the proposed method. The effectiveness and superiority for detecting ventricular ectopic beats (VEB) and supraventricular ectopic beats (SVEB) events are demonstrated. The proposed mechanism of fine-tuning the GCNN to TDCNN improves the efficiency for training patient-specific CNN classifier. Because of the computational efficiency of fine-tuning, ECG diagnosis and heart monitoring can be easily implemented with popular wearable devices in practice. (C) 2018 Elsevier B.V. All rights reserved.