Sketch-based cross-domain image retrieval (SBIR) uses a sketch as query to retrieve the most similar image from the color image database. In this study, in order to better fuse the features from sketch and color image, a hybrid cross-domain joint network for sketch-based image retrieval was proposed, consisting of a sketch feature extraction branch and a color image heterogeneous feature fusion network branch. The network extracts the feature representations of sketch, positive and negative color image, and corresponding edge outline, and fuses the features of the color image and its sketch approximation (the edge outline of the color image) as the color image feature, which bridges the cross-domain gap between sketches and images. The network model parameters and network structure were further explored to optimize the purposed algorithm. Experiment on Flickr15K dataset shows that the proposed method performed better than other advanced image retrieval methods. The mean average retrieval accuracy of the proposed method was 0.584 8, which was 0.052 2 higher than the optimal value in other methods.