To study the influence of boundary effect on phonon heat transport in nanoscale material, the phonon heat transport in silicon dioxide material for rectangular geometry is simulated, in which the effect of boundary scattering on phonon mean free path is considered. The results show that the effective thermal conductivity in nanoscale material reduces more obviously for diffuse reflection than the case of specular reflection, and the effect of boundary exists at the boundary. The effective thermal conductivity reduces with the decrease of specularity factor. The increasing of boundary roughness in nanoscale material, which enhances the thermal resistance of boundary, is in favor of the reduction of the thermal conductivity.