To investigate the main skeleton structure and void ratio in mineral aggregate (VMA) of porous asphalt concrete (PAC), according to the porosity of PAC, all mineral aggregates were divided into four parts: larger size aggregate (LSA), dominant aggregate size range (DASR), potentially disruption aggregate (PDA), and interstitial component aggregate (ICA). The main skeleton void structure formed by DASR was analyzed, the particle size range of each aggregate part was determined, and the calculation method of the void ratio in main skeleton was proposed. Based on the volume method, the physical model for VMA of PAC was developed by introducing the filling mechanism of ICA, interference of PDA, and substitution effect of LSA, which had a good prediction effect compared with compaction test results of PAC-13 and PAC-10. Results show that for the main skeleton void structure formed by DASR, ICA mainly played the role of filling the skeleton voids. The interference of PDA though occupied part of the skeleton voids formed by DASR, increased the main skeleton voids to a certain extent. LSA with larger particle size also increased the void ratio of the main skeleton structure after replacing part of DASR. LSA, PDA, and ICA had different directionality effects on the void ratio of main skeleton formed by DASR. In the initial aggregate grading design of PAC, there is no need to prepare test specimens, and the corresponding VMA can be predicted by determining the critical particle size of the four aggregate parts according to the grading curves. It is convenient to determine the best gradation by using this model, which greatly reduces the experimental workload for PAC mixture design. In conclusion, the proposed model is reasonable and effective, and has a good prediction effect.