In order to solve the problem that the relative orientation between depth camera and inertial measurement unit is difficult to measure directly, a non-contact calibration method based on the construction of displacement vectors by sensing the same hand motion is proposed to compute the relative orientation. Firstly, the relative orientation calibration problem of a depth camera and an inertial measurement unit with time-varying pose relationship is described and analyzed. Then the depth camera and the inertial measurement unit are used to simultaneously capture the movement of a hand swinging in arbitrary direction in the space, to construct displacement vectors. Finally, leveraging these displacement vectors, a model is built up based on the rotation invariance principle of a rigid body, and then solved with the least square method to obtain the calibration result. To verify the accuracy and effectiveness of the proposed method, in the one hand, a comparison between the measured data and the white noise simulation data is conducted. The simulation result shows that the relative orientation deviation after calibration is less than ±4°. On the other hand, an experiment to capture the human arm motion with a depth camera and an inertial measurement unit is also conducted. The experimental result shows that parameters of the human arm could be only correctly reflected after calibration. The calibration method presented in this paper is simple in principle, easy to operate, and without requirement of contact measurement or other auxiliary equipment, which is applicable to sensor calibration in scenarios such as robot remote control and motion sensing games.