Abstract: The decision surface of neural networks is high-dimensional, and the existing analytical methods are difficult to study its shape characteristics and inefficient. Traditional adversarial training is easily affected by uneven samples, which is difficult to reduce vulnerability. In order to explore the source of vulnerability efficiently, this paper proposes a sampling method based on decision boundary. It avoided using high-dimensional function to analyze decision boundary by Monte Carlo method. In order to reduce vulnerability, this paper proposed a new adversarial training method, which generated adversarial samples close to decision boundary through different strategies to achieve data augmentation. The experimental results show that singularity is one of the potential sources of vulnerability and the improved method can effectively improve the robustness.