To detect the potential risks of drainage pipelines and accurately grasp the pipeline conditions, a data-driven defect diagnosis model was established by combining optimized extreme learning machine (ELM) neural network and closed circuit television (CCTV) inspection. Genetic algorithm (GA) was adopted to optimize the input weight matrix and the hidden layer offset of ELM neural network, which helps to solve the problems of unstable output and low classification accuracy of ELM neural network caused by random generation of network parameters. Taking the drainage pipeline dataset from Yangshan Free Trade Port Area in Shanghai as an example, the proposed GA-ELM model was conducted to identify and diagnose major structural defects, such as pipe rupture, disconnect, and leakage. The results of the GA-ELM model were compared with those of ELM model on the same dataset. It shows that the GA-ELM model achieved better classification performance by utilizing same neuron nodes in the hidden layer, and the optimization of parameters improved the fitting capability and generalization ability of the ELM model. Therefore, the proposed method is applicable to defect diagnosis and evaluation of urban drainage pipes and can provide a technical basis for the formulation of drainage network maintenance plan and repair plan.