Track stiffness irregularity, indistinguishable on rail surface, causes dynamic impact or track deformation when trains running through, which greatly affects the operation safety and stability of the system. In view of this practical issue, a determination method of track stiffness irregularity was analytically deduced. Then, a high-speed train-track-bridge dynamic interaction model was established based on the train-track-bridge dynamic interaction theory. On this basis, the influence of normal track stiffness irregularity on the dynamic behaviors of the system was investigated in time and frequency domains. Finally, taking the fastener failure as an example, the influence of abnormal track stiffness irregularity on the dynamic performance of the system was studied. Results show that track stiffness had obvious effect on the vibrations of the system. Wheel-rail force, wheelset acceleration, and vertical acceleration of frame were obviously influenced by fastener spacing and length of slab. Pitch acceleration of frame was most sensitive to track stiffness irregularity. Considering structural elasticity, track vibrations at the slab-edge were much larger than those at the mid-span. The amplitude ratio of rail acceleration at different locations was 1.17, while that of slab acceleration reached 2.2. Normal track stiffness irregularity caused periodical vibrations, which might lead to the resonance of the system and speed up the structure damage, while abnormal track stiffness irregularity caused wheel-rail impact and affected the running safety of the system in serious situations. When the train ran at 250-350 km/h, the allowable number of failure fastener was one.