To study the influences of roughness, gravel content, and normal stress on the shear mechanical properties of the interface between limestone spoil mixture and concrete, a series of interface shear tests were carried out on limestone spoil mixtures with four types of gravel content and concrete surfaces with five types of roughness under different normal stress conditions by using a new large-scale direct shear apparatus. The influences of roughness, gravel content, and normal stress on the shear strength of the interface were investigated, and the internal relationships between roughness, gravel content, normal stress, and the shear mechanical properties of the interface, as well as the mechanism of shear strength were revealed. Test results show that under the same normal stress condition, as the roughness increased, the shear strength of the interface increased and then decreased, and the rough interface significantly increased the degree of dilation of the interface, but with the increase in normal stress, the influence of roughness on the shear strength and normal deformation of the interface was weakened. Under the condition of same normal stress, with the increase in gravel content, the variation of the shear strength of the interface was closely related to the magnitude of the normal stress. Besides, the shear strength of the interface was highly consistent with the Mohr-Coulomb strength criterion, and the influence of roughness and gravel content on the apparent cohesion of the interface was more significant than that on the internal friction angle of the interface. To a great extent, rational roughness and gravel content can improve the shear strength of the interface between spoil mixture and concrete.