Large Language Models for Automated Requirement Classification

Requirement engineering is the first phase in the development of a new software system. It consists of eliciting, analysing and managing stakeholder needs and expectations. A central activity is requirements classification, which is often described as repetitive, time-consuming and prone to errors that may propagate into later design and implementation decisions. Given recent progress in Large Language Models (LLMs) and their strong natural-language capabilities, an open question is whether they can reliably support engineers in requirements classification. Key concerns include robustness and reproducibility, as LLM outputs heavily depend on prompt design, model configuration, and execution strategy.

This project investigates whether LLMs can assist with automated requirement classification by benchmarking multiple ChatGPT model versions against expert-labeled ground truth. A web-based prototype platform is implemented to classify requirements using different prompt strategies and execution modes, and to evaluate predictions using standard classification metrics. Inspired by prior work on prompt engineering for Requirements Engineering tasks, the evaluation compares a baseline manually written prompt with an optimised prompt obtained through a prompt optimization step. In addition, the study contrasts single-requirement classification with batch processing of 50 requirements per request.

The evaluation is conducted on 625 labeled requirements derived from the PROMISE_exp dataset. Results show that classification performance varies across model versions and is sensitive to prompt formulation. The optimised prompt consistently improves performance over the baseline, while batch processing achieves comparable results with significantly reduced operational cost. These findings suggest that LLMs are best suited as decision-support tools in a human-in-the-loop workflow rather than as fully autonomous classifiers.