Abstract: Malicious traffic detection technology based on deep learning is widely used in intrusion detection systems in realistic scenarios. In order to ensure the effectiveness of intrusion detection systems in adversarial attack scenarios, this paper proposes an adversarial training algorithm based on deep reinforcement learning from the perspective of model optimization. The gradient projection descent algorithm was used to generate nonlinear disturbances in the threshold range. A deep reinforcement learning model was used to optimize the adversarial samples through interactive search with the target model. The model was trained with the optimal adversarial samples to improve its robustness.