To enhance the stability and speed of surface crack detection, a detection algorithm combining PeleeNet and YOLOv3 is proposed. PeleeNet framework is used to replace Darknet-53 framework of YOLOv3 so as to effectively integrate different local features and improve the detection rate. The feature attention module is integrated into the PeleeNet’s framework to highlight the saliency of the crack detection in the image, and through the receptive field module RFB broaden the effective field of view of the network, and increase the detection accuracy of small targets. Instead of the standard convolution, the depth separable convolution is employed to reduce the amount of parameter calculation in the feature pyramid network. Then, the CIoU loss function is introduced to strengthen the classification and regression accuracy of the model. The experimental results on the fracture data set show that AP₅₀ and AP₇₅ reach 97.68% and 77.87% respectively, 8.4% and 12.4% higher than the original YOLOv3. Meanwhile, the detection speed reaches 30 frames per second with the size of model parameters being only 30% of YOLOv3. As can be seen, the PeleeNet_yolov3 lightweight model proposed has produced an obvious effect on the detection of crack targets, with a small amount of calculations and parameters involved. Suitable for mobile terminal system, the study presents a great value of application especially for small volume, low power consumption and low computing power computing platforms.