For the edges between objects and background in an image are intertwined or their common boundaries are vague as well as the textures of objects and background are similar, a new method based on graph theory and closed-form solution was proposed. First, it uses closed-form solution to initially separate the objects from background roughly, then, to extract the detailed information of inter objects, it applies an improved graph-based algorithm to obtain the final image segmentation results. The test results show that the algorithm of matting avoids aliasing of foreground and background and the improved graph-based algorithm increases segmentation accuracy by 6％~12％ effectively. Compared to the traditional algorithms such as region merging, ordinary graph, and thresholding, the new algorithm has the better accuracy and effect, therefore it has the significant superiority.