In order to expand the application range of the virtual chamber measurement system, improve the measurement precision in actual measurement, a mathematical method of adaptive filtering and wiener filtering is used to analyze the inherent defects of the measurement system based on traditional ANS (automatic noise suppression), and point out the main factor restricted the application range and measurement precision is the existence of source signal component which is detected in the reference channel. According to the analysis result, the measurement system is rebuilt, coming up an ANS based on spectral estimation. Analyzing the mixed signal detected in the main channel, the remodeled system can get the PSD (power spectrum density) of radiation signal and noise signal .According to the frequency of the peak point in the PSD, notch filter has been designed to filter the radiation signal component captured by reference channel, then the system meets its expectation. The simulation and experiment results show that this method shows good performance in the high SNR (signal noise ratio) measurement circumstance as well as low SNR, which means that extracting the radiation signal of equipment under test exactly, high noise suppression ability and environmental adaptability of virtual chamber.