Due to the periodic nonlinear error, heterodyne laser interferometer cannot meet the requirements of the sub-nanometer and even picometer measurement accuracy of the next generation of ultra-precision equipment manufacturing and major scientific engineering. Aiming at the problem, this paper analyzes two types of periodic nonlinear errors in heterodyne laser interferometer and investigates their compensation methods. Results show that the first type of periodic nonlinear error is caused by optical mixing due to the incomplete separation of dual-frequency lasers, whose amplitude ranges from several nanometers to tens of nanometers. The second type of periodic nonlinear error is induced by multi-order Doppler frequency shift (DFS) ghost beam generated by the ghost reflection of measurement beam at the optical interface, whose amplitude ranges from several picometers to several nanometers. For the first type of periodic nonlinear errors, the current nonlinear error compensation methods, such as ellipse fitting method, can suppress them to 0.1 nm level. In particular, the spatially separated heterodyne laser interferometers proposed in recent years can completely eliminate the first type of nonlinear error in principle. As for the second type of error, by reducing ghost reflectivity and spatial filter, the error can be reduced to tens of picometers or hundreds of picometers, while the residual error is still too large to meet the accuracy requirements of picometer measurement. Thus, it is urgent to develop new error suppression or compensation technologies.