AI Development Needs an Evaluation Harness: Practical Tools for Assessing LLM Outputs

1. Looking functional does not mean avoiding regressions

Rewrite a prompt, try a few examples by hand, decide that the results look good, and ship. This approach is fast initially, but has a structural weakness: it fails to reveal quiet regressions after model upgrades or small prompt changes. Practical guides to evaluation tools repeatedly describe the same problem. Changing a prompt or model can shift the entire output distribution in ways that are difficult to see, and manual review cannot detect those shifts at scale.

2. A concrete example: evaluations in a CI pipeline

Rather than stopping at the abstract advice to have an evaluation harness, consider how one fits into a workflow. The open-source evaluation tool promptfoo defines prompts, test cases, and pass/fail conditions in YAML files. A single command, promptfoo eval, can score multiple prompt and model combinations side by side.

The practical point is that the tool can return a nonzero exit code when evaluation results fall below a threshold. This allows it to fit into a conventional CI/CD pipeline such as GitHub Actions just like other software tests. A pull request that changes a prompt or model can be blocked from merging unless its scores on the existing regression suite meet the required standard. OpenAI was reported to have acquired the tool in March 2026, with development continuing as open source under the MIT license.

3. The minimum elements of an evaluation harness

1. A regression test set: Repeatable test cases built from input patterns that caused problems in the past.
2. Quantitative scores: Automated scoring of qualities such as accuracy, consistency, and harmfulness.
3. Human sampling reviews: Regular human checks for quality degradation that numerical evaluations miss.
4. Release gates: Block merging or deployment when a change falls below a threshold, using an ordinary CI mechanism such as the one described above.

4. The connection to AIQDD

This is also a core principle of AIQDD, or AI Quality-Driven Development, as practiced by TIP. Code written by agents and outputs generated by models need continuous validation through an automated evaluation harness, rather than relying solely on people inspecting them. So far, we have found no other way to maintain quality without sacrificing speed. The practical point is that evaluation does not have to be a large undertaking requiring special infrastructure: it can be incorporated into an existing CI pipeline just like conventional test code. Evaluating AI outputs and testing software do not need to be treated as separate disciplines.