The traditional interacting multiple model (IMM) algorithm usually models the mode evolutions as Markov processes with constant mode transition matrix and leaves the correlative information among sub-filters out. However, because of the complexity of the practical application, the traditional IMM algorithm is unsuitable in aircraft tracking. To solve these problems, an optimal mode transition matrix IMM algorithm（OMTM-IMM）is presented. The new algorithm uses the linear minimum variance theory to calculate the optimal mode transition matrix according to the correlations between sub-filters. In this case, the new matrix further approaches the truth one, and the estimation accuracy can be improved. This conclusion can be support by the following theoretical derivation and simulations in aircraft tracking.