To characterize the mechanical behaviors of porous asphalt mixtures under loading at normal and low temperatures, a viscoelastic-plastic damage model, which includes Weibull damage function, Generalized Maxwell and Drucker-Prager model, was presented based on incremental constitutive equation. Experimental data from uniaxial compressive strength tests conducted at different strain rate and different temperature were used to validate the efficacy of the model, and the change rules of model parameters in different temperatures and loading rates were investigated. The test and analysis results show that parameters in viscoelastic model begin to behave like the elastic materials. The volume modulus and shear modulus also show obvious viscoelastic properties along with the changes of temperature and load rate. Additionally the strain plastic strain initializes keep almost same to the critical damage strain, which accommodate to the presented hypothesis. The damage curves of porous asphalt mixtures can also reveal the effects of different temperature and load rate. This model can successfully characterize the damage and mechanical behaviors of porous asphalt mixtures at normal and low temperatures.