This paper introduces the harmony search theory to propose a novel particle filter, and a visual tracking based on harmony search particle filter which can combine the current observation with history information to achieve a robust performance. Firstly, the importance sampling function is modified using such conceptions in harmony search theory as memory consideration, genetic variation, random variation and the current observation. These improve the robustness on system state transition matrix. Secondly, parameters of harmony search are optimized to balance the demand on timeliness and accuracy. Moreover, the weight of particle is compensated to further accommodate the Bayesian estimation.Simulations show that the optimized harmony search parameters are more suitable for harmony search particle filter than common parameters. Compared with classic visual tracking algorithms, the proposed algorithm demonstrates more accurate visual tracking ability under complex environments such as illumination changing and occlusion.