This paper analyzes the influence of urban outdoor leisure sport space (OLSS) factors on leisurely physical activities (LPA) between genders by a survey in Harbin. A model about the relationship of OLSS factors and LPA was established by the Structural Equation Model (SEM). Using simultaneous analysis of several groups of SEM, the genders as moderator variables were introduced into the model to test the influence of OLSS factors on LPA between genders and the corresponding difference. Result shows that in the male group, facilities, accessibility, amenities, aesthetics, maintenance, and safety had significant impacts on the frequency of LPA. There was no OLSS factor that affected the duration of LPA, and facilities had a significantly greater impact on frequency than accessibility. In the female group, factors that impacted frequency of LPA were identified as facilities, amenities, aesthetics, maintenance, and safety. The two factors which impacted the duration of LPA were facilities and physical environment, and physical environment had a significantly greater effect on duration than facilities. By comparing the two groups, accessibility had significant influence on the frequency of males LPA and physical environment greatly affected the duration of females LPA. Amenities had a greater impact on the physical environment in the female group. The influence and disparities of those relationships in different gender groups were analyzed and interpreted, and the suggestions for designing OLSS considering different gender characteristics were proposed.