Nanofiltration technology offers an effective solution for the separation and resource recovery of heavy metals in industrial wastewater. However, the physicochemical properties of nanofiltration membranes suitable for heavy metal separation remain unclear, making it difficult to prepare high-performance nanofiltration membranes for heavy metal separation. This paper developed a machine learning prediction model incorporating membrane characteristics, operating parameters, and solution properties. Through a comparison of ten models, the XGBoost model yielded R² values of 0.98 and 0.92 for membrane flux and heavy metal rejection rate, respectively. The SHAP analysis results revealed that water contact angle, molecular weight cut-off, and membrane surface potential were the key determinants of performance. Enhancing hydrophilicity, moderately increasing the molecular weight cut-off, and imparting more positive charges on the membrane surface collectively enhanced membrane flux. In contrast, heavy metal rejection rate was governed by the combined effects of size exclusion and electrostatic repulsion. Based on the high-accuracy prediction model, an NSGA-Ⅱ bi-objective optimization algorithm was constructed. The results show that within the ranges of water contact angle from 60° to 66°, molecular weight cut-off from 200 to 412 u, and membrane surface potential from 47 to 73 mV, nanofiltration membranes can achieve efficient treatment of 10 kinds of heavy metal ions, with rejection rate exceeding 99% and membrane flux ranging from 25 to 30 L/(m²·h). The nanofiltration membrane optimization framework developed in this paper provides both theoretical and technical support for the development and application of high-performance nanofiltration materials.