Abstract: Natural language has been widely used as one form of software requirements as it is easy to understand. But natural language requirements are prone to defects. At present, applying natural language processing techniques on requirement defects has gradually become a research hotspot. However, unlike other fields having a large number of publicly available datasets, in the field of software engineering, there is still a lack of suitable datasets and methods to evaluate whether datasets are sufficient for helping perform tasks such as natural language defect detection. Aiming at the task of requirement defect detection, we propose an evaluation method and quantitative metric model for corresponding dataset, and designe a rule-based evaluation framework. We experimented with existing public requirement dataset, and conducted statistics based on quantitative metrics.