To accurately forecast rogue waves and avoid the great harm to the safety of the buildings and people on the sea, by utilizing the compact wavelet neural network model, combined with the wave height test data in a three-dimensional model of reefs established based on the measured data of the topography of a reef, time series of three typical wave heights were selected from experimental data to realize the prediction of wave data containing rogue waves against conventional waves, the prediction of wave data containing approximate rogue waves against rogue waves, and the prediction of conventional waves against wave data containing rogue waves. In order to verify the accuracy of the wavelet neural network model, the conventional neural network BP model was used to predict the time series of the three typical wave heights under the same conditions. Finally, the accuracy of the two neural network prediction results was compared. Results show that the wavelet neural network could capture the rogue wave emergencies better. For the overall prediction accuracy of wave surface and the prediction accuracy of rogue waves in three working conditions, the prediction model of wavelet neural network was higher than that of BP neural network, and the predicted wave height curves had better fitting effect with the actual wave height curves. If there were rogue wave features in the neural network training samples, the prediction accuracy of future rogue waves would be further improved. This research has certain application value for the risk warning of rogue waves in ships or marine engineering.