Arrhythmia is characterized by irregular heartbeats, and arrhythmia classification plays a key role in early prevention and diagnosis of cardiovascular diseases. In order to improve the accuracy and speed of arrhythmia classification and realize the automatic recognition of arrhythmia types, a seven-layer hybrid model based on convolutional neural network (CNN) was proposed. To maintain the integrity of the beats, the electrocardiogram signal was dynamically segmented according to the R-R interval to obtain different lengths of heartbeats. The local features of the heartbeats were extracted through the sliding of the convolution kernel of convolution layer, and the average pooling layer performed down-sampling to reduce the dimensionality of the features. The spatial pyramid pooling (SPP) layer extracted the beat features with different pooling sizes. Input features of different lengths were fused by SPP layer to obtain output features of the same length. Extreme learning machine (ELM) as a classifier could improve the speed of classification and shorten the training time. The MIT-BIH arrhythmia database (MITDB) and ten-fold cross-validation method were adopted to complete four classification experiments of arrhythmia. The overall accuracy, sensitivity, specificity, and precision of the classification results in the test set reached 99.16%, 99.85%, 98.89%, and 99.85%, respectively. In the same software environment, the accuracy and training time of hybrid model and single model were verified, and results show that the hybrid model achieved higher accuracy with less training time, which provides a feasible scheme for quickly and accurately identifying the types of arrhythmia.