The indoor physical environment of buildings such as light environment, wind environment, and temperature and humidity field has great impact on the life quality of users, so the design of indoor physical environment is especially important. Surrogate model is an effective rapid prediction method of building performance, which can help architects in selecting or iteratively adjusting the plan at the initial stage of design. However, most existing surrogate models focus on calculating single valued data such as energy consumption and daylighting coefficient, lacking of calculation methods for “matrix” data (i.e. cloud map) such as illuminance distribution, wind field, and temperature and humidity field. To this end, a building performance prediction method was proposed combined with the conditional generative adversarial network (CGAN). Based on the pix2pix algorithm, the CGAN model of cloud images of illuminance and wind field was constructed, and the mapping relationship between the images was established. The performance data of the corresponding plane were generated by inputting the building outline, so as to rapidly predict the building performance and improve the efficiency of the initial design of the building. In order to evaluate the accuracy of the prediction results, we established an evaluation system based on the channel histogram method and the grayscale value method, which was applied for similarity verification. Results show that the trained architectural CGAN model could quickly draw cloud images within 1 s, which was much faster than traditional simulation software, and the prediction results were highly consistent with the simulation calculation results. The similarity between the results reached 93.82% (illuminance) and 82.99% (wind field), which proved the effectiveness of the prediction method of building indoor matrix data based on CGAN model.