To design low energy return trajectory for unmanned lunar probe, the dynamic model of the probe is developed under the elliptical four body problem in consideration of the effect of sun's gravitation and lunar elliptical motion to the probe's orbit. The existence and the dynamical characteristics of the return trajectory are analyzed. To deal with the strong nonlinearities of the dynamic model and the local convergence in the optimal process, an adaptive genetic algorithm is proposed which can adjust the self-evolution parameters according to the fitness of the population to improve the effectiveness of the evolution of the population to the global optimal point as well as to reduce the computation burden. According to the simulation results, the algorithm works well in optimizing the energy needed for the lunar probe to return to Earth, which is only 75% of that in the traditional hyperbolic matching method.