The claw compressor is the most promising hydrogen circulating pump type due to its remarkable advantages of compact structure, dry oil-free and high reliability. However, conventional claw hydrogen circulating pumps suffer from severe gas leakage between two claw rotors due to point to point meshing, resulting in reduced volumetric efficiency of the pump and limiting the development of the hydrogen circulation pump. In order to minimize gas leakage between two claw rotors and improve the volumetric efficiency of pumps, this study proposed a novel twisting meshing structure adopting the circular arcs, high-order curves and their conjugate curves to instead of pitch circular arcs of conventional claw rotors. This leads to the development of a novel gear-claw rotor, effectively solving the issue of gas leakage between two claw rotors. Simultaneously, a geometric model of the novel gear-claw rotor was established, and the equations of rotor profiles were deduced. The transient flow of the internal gas in the hydrogen circulating pump with complex geometric boundaries was simulated in the working process. A comparative analysis was conducted between the pressure distributions and transient flow characteristics inside the working chamber of the conventional pump and the proposed gear-claw pumps. Additionally, a performance test bench for the claw hydrogen circulating pump was built to validate the accuracy of the numerical simulation results. The results indicated that compared with conventional claw rotors, the gas leakage velocity between the gear-claw rotors was reduced by 31.42% and 33.09% during the compression and discharge process, respectively, and its volumetric efficiency was increased by 10.92%.