To meet the requirement of satellite imaging during maneuver, the mixed control moment gyroscopes and flywheel are used as the actuator to provide large and precise control torque. Due to the fact that the installation deviation of these actuators will reduce the attitude control accuracy of satellite, an adaptive control law is designed to deal with this issue. Based on the dynamic model of a satellite with a cluster of single-gimbal variable-speed control moment gyros, the control model with the consideration of installation deviation is derived. Based on the kinematic equation described by modified Rodrigues parameters, a multi-input multi-output adaptive tracking control law is designed to estimate the installation deviation of the actuators and the inertia of satellite, and the corresponding control compensation is adopted to improve the control accuracy. The singularity phenomenon of the tracking control law is avoided by using smooth projector principle, and the stability of the controlled system is analyzed via Lyapunov theory. Simulation results show that the proposed adaptive controller enables the satellite fast maneuver with high control accuracy, and the accuracy of the tracking control can be improved by two orders of magnitude.