Automatic Validation of Use Case Descriptions in terms of quality criteria and bad smells via NLP models and LLMs

Ensuring the quality of use case descriptions is a critical task in software engineering, as poor quality artifacts can lead to ambiguity, miscommunication, and design flaws. Current quality assessment methods are largely manual, time consuming, and prone to inconsistency. Existing approaches are limited in scope and rarely address the nuanced quality criteria or recurring issues---commonly known as “bad smells”---in a systematic way. This paper presents an observational study that explores the feasibility of detecting quality criteria violations and bad smells in use case descriptions using a combination of Natural Language Processing (NLP) techniques and Large Language Models (LLMs). The system operates based on a predefined set of quality criteria drawn from established literature and practitioner guidelines. It targets five key fields in use case descriptions: Name, Actors, Postcondition, Standard Procedure, and Extensions. Rather than seeking to generalize, this study serves as a first step toward understanding whether such detection can be performed using language models. Evaluation results show promising levels of accuracy, precision, recall, and F1 scores. A comparative analysis of GPT-4o and o1 highlights trade offs in output quality, runtime, and cost, with GPT-4o emerging as the more practical choice. While the system provides consistent results to the extent permitted by the LLMs used, the findings suggest that criteria based, LLM and NLP models supported quality assessment of use case descriptions is both feasible and worthy of further exploration.