A new and universal scheme for passive decoy-state quantum key distribution (QKD) with the two-mode state source is proposed. As one of the two-mode, the trigger state can obtain four types of detection events after splitting and detecting at the transmitter. Based on those events, the other mode as the signal state is divided into four sets of pulses which can be used to estimate parameters and extract key. Besides, the performance analysis is carried out on the scheme with the heralded single-photon source (HSPS) and the heralded pair coherent state (HPCS). The impact of different detection efficiency is discussed. Moreover, the statistical fluctuation in the practical system is numerically studied. Our simulation results show that the performance of our scheme is superior to the existing three-intensity decoy-state QKD schemes with different sources in terms of bit error rate and secure transmission distance (up to 198.6 km). The performance of using HPCS is better than that of using HSPS, and the key generation rate increases with the rising of detection efficiency at the transmitter. Considering the statistical fluctuation, the efficiency of HPCS is also better than HSPS, and the maximum secure distance of our scheme can reach 164 km when the data length is 10⁹. Furthermore, our scheme only needs to use a single intensity pulse, which reduces the difficulty of the system implementation while improving the system performance. It has certain reference value to the implementation of QKD system.