To realize the lossless recovery of non-stationary signal in complicated noise environment, a novel design method based on fractional Fourier transform of time-frequency filter is proposed, in which the time-frequency distribution of incident signal is obtained by Gabor transform first, and then based on support vector machine (SVM) and technique of image segmentation, the regions of signal and noise on the time-frequency plane are separated and the optimal separating line is drawn, finally the order number and transfer function of the time-frequency filter can be determined by the optimal separating line equation. For the case of linearly inseparable signal and noise time-frequency distribution, the piecewise linear fitting based on global least square criterion is performed to the separating curved line, and the parallel filter banks are constructed from the linear fitting equation. To meet the real-time requirement in engineering application, the computational complexity was optimized, and the simulation results demonstrated the validity of this method.