Existing speed limit strategies cannot reflect the effects of geometric alignments and the road friction condition on vehicle lateral stability precisely. Regarding this issue, the lateral instability of vehicle running on circular curve segment was decoupled into three instability modes. A precise method for calculating the safety boundary of each mode was presented, and the maximum speed which ensures vehicle lateral stability was deduced. Firstly, the drawbacks of the traditional lateral stability indicator, i.e., the side friction factor, were analyzed. The instability status was decoupled as steering instability, losing track-holding capacity, and rollover, and the corresponding safety evaluation indices were proposed. Secondly, a 7 degree-of-freedom (DOF) nonlinear vehicle-circular highway segment coupling model was developed, with which the computational models within the safety boundaries of safety evaluation indices were deduced. Subsequently, the accuracy of the safety boundaries were validated in various road friction conditions through the employment of Carsim. Finally, based on the 7 DOF nonlinear vehicle-road coupling model and the computational models of safety boundaries, the critical safe speeds for circular curves were deduced using MATLAB/Simulink, and the comparisons among the proposed method, the operating speed, and the design speed were made. The results indicate that design speed limit is too conservative on dry and wet pavement, while operating speed often exceeds critical safe speed on wet and icy pavement, which may cause safety issues. The speed limit strategy proposed in this paper fully considers the non-linearity of vehicle’s lateral motion, and could be used as a complement for other speed limit strategies for mountain highway which takes driving expectancy and efficiency into account.