To improve the performance of particle filter,ensemble Kalman filter is proposed to construct proposal distribution. And an adaptive fusion model is applied for object tracking. Using ensemble Kalman analysis to build the posterior probability distribution by integrating latest observation information. New particles are resampled from the new proposal distribution. In the tracking process, color model and shape model are fused and updated adaptively. Experimental results show the new proposal distribution can reduce the root mean square error more effectively than traditional particle filter and extended Kalman particle filter. The adaptive fusion model is more stable than single color model. The new proposal and adaptive fusion model can enhance the estimation accuracy and improve the stability of the object tracking.