To accurately evaluate the wobble error of the rotary axis system, the relationships among the wobble error of the rotary axis system, installation errors of the reflected mirror and of the autocollimator, and the readouts of the autocollimator are established firstly. Installation errors of the reflected mirror are referred to be the first harmonic in the quadrature phase of the autocollimator readouts, and then the formation mechanisms of the first harmonic resonance are analyzed, which is represented as the first harmonic in the same phase. A separation method to handle the first harmonics in the same phase and in the quadrature phase are designed for the first harmonics in the x-and the y-direction readouts of the autocollimator. This method eliminates the first harmonics caused by the installation errors of the reflected mirror, and retains the first harmonic resonance in the rotary error. As a result, the data processing method of the wobble error of the rotary axis system is improved. Finally, by changing the installation errors of the reflected mirror, the wobble errors of the same rotary axis system are measured two times. It is proved that this separation method is correct since the first harmonic resonance errors from the two groups of the measured data are consistent. Meanwhile, it only eliminates the first harmonic caused by the installation errors of the reflected mirror, and retains the first harmonic resonance errors in the rotary errors. The experimental results show that the method of data processing for wobble error of the rotary axis system is more reasonable and accurate.