To solve the problems of imperfect dynamic model and difficulty in obtaining nonlinear dynamic characteristics of non-circular planetary gear systems, a dynamic modeling method of non-circular planetary gear systems is proposed. The focus is on the dynamic response mechanism of the system under time-varying parameter excitation, aiming to study the nonlinear dynamic characteristics of non-circular planetary gear systems. In order to accurately analyze the nonlinear vibration of the system, the backlash function of non-circular gear is fitted. By considering the combined effects of tooth surface friction, time-varying meshing stiffness, viscoelastic damping and static transfer error, the variable coupling of nonlinear equations is eliminated by introducing relative displacement coordinates. On this basis, the dynamic model of non-circular planetary gear systems is established, and the fourth-order variable step Runge-Kutta numerical method is used to solve the nonlinear dynamic equations of the system. Bifurcation diagram, time domain diagram, phase trajectory and Poincaré map are obtained to reveal the distribution pattern of system’s the dynamic response under the influence of control parameters such as damping, tooth surface friction and time-varying meshing stiffness. The results show that with the different values of excitation parameters, the system presents a transition state between chaotic and periodic motion depending on the values of various excitation parameters. By selecting appropriate excitation parameters, the time interval between chaos and periodic motion can be reduced, leading to a quicker transition to a stable motion state. The research results can provide a theoretical basis for suppressing nonlinear vibrations of non-circular planetary gear systems and predicting the dynamic behavior of the system.