The lever arm effect superimposed on the accelerometer sensitive axis of the attitude and heading reference system (AHRS) under vibration environment will greatly affect the accuracy of the attitude measurement based on the accelerometer. This paper analyzes the low-pass filtering method and the mechanical compensation method in the lever arm effect processing algorithms. The low pass filtering method will have rectification error residual. The compensation of the centripetal acceleration term in the lever arm effect by the mechanical compensation method is limited by the gyro measurement accuracy. Meanwhile, the difference operation of angular velocity will greatly enlarge the compensation error of tangential acceleration term in lever arm effect. Aiming at the shortcomings of the low-pass filtering method and the mechanical compensation method, this paper proposes an improved compensation method of lever-arm effect. For the centripetal acceleration term, the angular frequency threshold is set as decision value. When the angular frequency is lower than the given angular frequency threshold, the low-pass filtering method is adopted; and when the angular frequency is higher than the given angular frequency threshold, the mechanical compensation+low-pass filtering method is adopted. For the tangential acceleration term, the angular acceleration can be solved analytically by multiple accelerometers, which can avoid the error amplification caused by differential operation. Through the theoretical derivation, analysis and simulation, the improved compensation method significantly improves the compensation precision of the lever arm effect, compared with the low-pass filtering method and the mechanical compensation method.