Rain weather will affect the quality of the outdoor images and blur and cover the image information. To improve the clarity of outdoor images affected by rain streaks, restore image feature information and improve the accuracy of many computer vision algorithms under rain weather condition, a single image rain streaks removal method based on the multi-scale convolutional neural network is proposed. Firstly, the structure of the multi-scale convolutional neural network is established and the method extracts information of images by multi-scale convolution. The information is utilized to remove the rain streaks and reconstruct images. Then, the method combines the features of rain which include low saturation and high intensity to train the network. The optimal parameters can be obtained in this way. Finally the convolutional neural network can remove the rain streaks in single images effectively. Experimental results show that the proposed method compared with the existing methods has better performance about rain streaks removal. It can better maintain the original information of the image and avoid blurring of the image. Meanwhile, using multiscale convolution to extract feature information can enrich the feature information and it is beneficial to improve the rain removal ability of the convolution neural network.