In order to improve the practicability of segmentation algorithm in weld defect detection, a lightweight weld defect evaluation network MYNet was proposed. In the network, the lightweight residual structure could reduce the amount of calculation of the model, the feature pyramid network (FPN) combined with multi-layer visual fusion mechanism could improve the segmentation ability of the network, and the parallel mask mechanism could obtain a fast and high-quality defect segmentation mask. The open source computer vision library platform OpenCV was introduced to calculate different defect areas by pixel threshold, and Tencent’s ultra-high-performance mobile platform reasoning framework was introduced to accelerate the forward reasoning speed of the model in the central processing unit. In this study, a digital artificial intelligence evaluation device was built with the ARM Cortex-A72 architecture as the control core, and a suitable lightweight 64-bit Linux system was deployed for defect detection to verify the feasibility of the proposed weld defect evaluation algorithm. Experimental results show that the model could effectively locate and learn different types of defect features. The network evaluated the defect area and location information with the accuracy of 94.64%. Compared with the MS R-CNN network which has high accuracy but requires a large amount of calculation, the accuracy of the proposed method was only reduced by 1.93%, while the parameter weight was only 1/14 of the MS R-CNN network, and the computing power required for network execution was lower. The lightweight residual structure increased the forward reasoning speed of the network by 309%, and it only took 1.7 s to complete the task of segmentation of weld instances on the low-cost hardware based on the advanced RISC machine (ARM) architecture. The method proposed in this paper can effectively learn and evaluate X-ray weld defect images, and the algorithm applied to the evaluation device reduced the cost of welding quality inspection.