To improve the quality of products in deep drawing process, the deep reinforcement learning method is used to optimize the blank holder force (BHF). A new BHF control model based on the integration of deep reinforcement learning and finite element simulation is proposed, and the BHF control strategy is optimized by combining the perception ability of deep neural network with the decision-making ability of reinforcement learning. The proposed control model uses the deep neural network to deal with huge state space and avoids the fitting of system dynamics. By utilizing a novel strategy network structure, the BHF control strategy is divided into global and local parts, and the control effect is improved. Meanwhile, the theoretical knowledge of BHF is used to initialize the replay experience, which improves the learning efficiency of deep reinforcement learning algorithm in BHF control tasks. Experiments show that the proposed BHF control model can optimize BHF control strategy more effectively than traditional deep reinforcement learning algorithm. The comprehensive performance of the proposed control model in three quality indicators (internal stress, thickness and material utilizing rate) is better than that of the traditional deep reinforcement learning algorithms.