The effect of screw pile and pile side soil is relatively complex. In order to study the vibration characteristics of screw pile under vertical load, the longitudinal vibration characteristics of screw pile in hysteretic damping foundation are studied from the three-dimensional fluctuation of pile side soil, considering the layered characteristics of soil and the vertical action of pile side soil. Based on the three-dimensional wave theory, the wave equation of pile side soil is established, and the vibration response solution of screw pile under the condition of complete coupling between pile and soil is obtained by Laplace transform and modified impedance function transfer method. The theoretical calculation results are compared with the field measured curves to prove the rationality of the screw pile-soil interaction model. The results show that compared with the assumption that the soil around the pile is homogeneous, considering the layered characteristics of the soil can fully consider the reflection and refraction of the stress wave at the layered interface, which is more in line with the actual situation. In the range of effective pile length, with the increase of pile length, the side friction resistance of pile side soil to screw pile is larger, and the damping effect is more obvious. The existence of the screw thread of the screw pile has a certain vibration reduction effect on the pile body, and with the increase of the outer diameter of the screw teeth, the amplitude level and resonance frequency of the complex stiffness curve of the pile top decrease significantly. The theoretical model can better simulate the interaction mechanism between the screw pile and the layered soil, and provide theoretical support for the application of the screw pile.