For the observability of nonlinear system,the observability matrix was derived using the Lie differentiation combining with the correlation theory of differential geometry,and an analytic method to measure the observability degree of nonlinear system was given.Then the method was applied in the observability analysis of deep space autonomous navigation system based on the Sun line-of-sight vector.The variations of observability degree for different orbital elements were investigated.Furthermore,in order to verify the relation between the observability degree and the estimation accuracy of the navigation system,the navigation process under different observability degrees was performed based on the extended Kalman filter.The method of the observability analysis proposed in this paper was demonstrated to be feasible from simulation results.