To improve the property of M-ary chaotic communication system and develop the chaotic synchronization technique of Duffing chaotic system, a quaternary chaotic digital communication system using Duffing oscillator is presented in this paper. The binary chaotic modulation method with Duffing oscillator was combined with Quadrature Amplitude Modulation (QAM) to realize quaternary chaotic modulation with Duffing oscillator. Meanwhile, according to the demodulation theory of Duffing oscillator, the zone partition was simplified, and the chaotic signal demodulator based on Duffing oscillators array as well as the quaternary chaotic digital receiver based on Duffing oscillator was constructed, which realized the quaternary chaotic digital communication system design based on Duffing oscillator. Simulation results show that the quaternary chaotic modulation could be achieved by the quaternary chaotic signal transmitter. The simplified zone partition could recognize the phase trajectory of the Duffing oscillator with easier realization of structure and algorithm. The quaternary chaotic digital receiver based on Duffing oscillator could realize noncoherent demodulation of the quaternary Duffing chaotic signal, and results show that the BER property of the chaotic system was better than that of the classical quaternary chaotic systems. Therefore, the quaternary chaotic digital communication system based on Duffing oscillator introduced in this paper simplifies the chaotic modulation method and noncoherent demodulation with Duffing oscillator, which provides reference for the construction of M-ary chaotic communication system.