Aiming at non-cooperative spacecraft with rectangular structures, a more simple and general relative attitude determination algorithm was deduced by expressing the relative attitude in the form of coordination transformation matrix. To verify the method, the image acquisition and processing system was designed based on MATLAB, and using STK the target simulator was built. Furthermore a ground test system was established, and the tests of flying around in the orbit plane and out of the orbit plane were performed. The results indicate that the relative attitude determination algorithm can derive the relative attitude of non-cooperative spacecraft exactly.