Nowadays, more and more MMICs (e.g., limiters and RF switches) have adopted both balanced and unbalanced test pad structures to address the challenging size restrictions and integration requirements of MMICs, which needs hybrid balanced-unbalanced RF probes for on-wafer test. However, the thru standard based on a single balanced or unbalanced structure cannot meet the impedance matching requirement of the hybrid RF probes at the same time, which leads to significant reduction of the calibration accuracy and fails to satisfy the requirement of MMIC test. Therefore, in this paper, the calibration error of hybrid balanced-unbalanced RF probes based on traditional SOLR calibration method is estimated, and an on-wafer test approach of MMICs based on hybrid balanced-unbalanced RF probes is proposed, which combines OSL second-order de-embedding technique with error correction and matrix transformation technique. Meanwhile, we designed an on-wafer test system which consists of a vector network analyzer and SG and GSG RF probes and so on to validate the solution. After vector error correction and de-embedding, the calibration reference plane can be accurately shifted to the probe tip, which greatly improves the test accuracy. In addition, an automatic test system was built by utilizing the object-oriented C# language to improve the efficiency. The program can realize fully automatic operation with instrument control, data acquisition, results correction, data analysis, and so on, which avoids human interference and solves the test efficiency problems faced with the wafer-scale test.