Resident travel information can reflect the activity routines of residents and urban traffic problems, which is an important basis for formulating transportation planning and management. Although the trajectory information acquired by GPS has a lot of spatio-temporal information, it cannot directly express transportation modes. Data processing and mining algorithms are needed to extract hidden knowledge to infer transportation modes, while recognition has great challenges due to the high degree of non-linearity and complexity of residents’ travel patterns. In this study, the advantages of deep learning were utilized to solve difficult calculation features or missing extraction features. After pre-processing of the trajectory information, kinematic features of the trajectory segments were calculated to form the input data. A method that combines convolutional neural network with gate recurrent unit was proposed to recognize transportation modes. By utilizing the advantages of convolutional neural networks, the deep features and the ability of gate recurrent unit were characterized to mine time series characteristics, improve the learning ability of nonlinear classification problems, and increase the accuracy of transportation modes recognition. In order to verify the effectiveness of the proposed method, separate convolutional neural network and gate recurrent unit were designed, which was tested and compared on the published GeoLife dataset. Experimental results show that although the proposed method only used four features, it still received well recognition results. Besides, the proposed method had better recognition performance than using a convolutional neural network and other classification methods.