In order to suppress the oscillation characteristics of relative navigation system caused by minimum measure resolution, an accurate microwave measure model is developed, and an adaptive step-size filtering algorithm is presented in the paper; Performance analysis and comparison between the adaptive step-size filtering algorithm and the low-bandwidth design algorithm widely used in the project is made. The theoretical comparison result indicates that the bias and random error increases inversely with the decreasing of filtering bandwidth, and convergence process becomes longer, but the adaptive step-size filtering algorithm largely reduces the performance loss. Simulation result shows that the relative velocity estimation error is 2E-3 m／s(3σ), which is reduced by one order of magnitude compared with the relative velocity estimation error of low bandwidth method, and can satisfy the mission requirements for high precision satellite formation.