In order to improve prediction accuracy of vehicle′s car-following behavior in the complex environment, a car-following behavior modeling method is proposed under an ensemble learning framework to integrate theory-driven model and data-driven model. Based on the stacking ensemble learning framework, a theory-driven model, intelligent driver model (IDM), and two data-driven models, long-short term memory (LSTM) network and gate recurrent unit (GRU) network considering the factors of vehicle platoon and surrounding driving condition, are selected as the first-level learning algorithms of car-following behavior features. Three linear and eight nonlinear regression methods are taken as the candidates of second-level learning algorithms to integrate the features of the first-level learner outputs. By comparing trajectory prediction accuracy calculated from real vehicle trajectory data, the optimal car-following model is determined. Results show that the data-driven car-following models, which take the variables of vehicle platoon and surrounding driving condition into account, have the higher trajectory prediction accuracy than the IDM. In most cases, the ensemble learning car-following models with the nonlinear second-level learning algorithm predicts vehicle trajectory in the higher accuracy than the IDM, the data-driven models and the ensemble learning car-following models with the linear second-level learning algorithm. The IDM-LSTM-stacking model, which uses the GBRT regression as second-level learning algorithm, and the IDM-GRU-stacking model, which uses the random forest regression as second-level learning algorithm, give the highest trajectory prediction accuracy. The stability of integrated car-following model under external disturbance is prior to single theory-driven model and data-driven model. The ensemble learning method provides a new approach for modeling driving behaviors.