LLM evals
LLM evals are systematic tests of a model or AI feature's output quality: a set of inputs, a definition of a good response, and a scoring method. They exist because you cannot tell from a handful of impressive examples whether something works reliably.
The hard part is scoring, not running. Some outputs are checkable — does the code compile, do the tests pass, is the JSON valid. Others need a rubric applied by a human or by another model, and model-graded evals inherit the grader's blind spots.
The discipline is the same as testing: a small suite you actually run beats a comprehensive one you designed and abandoned.
What this means in practice
Relevant if you are building something on top of a model rather than using a coding tool. If you are shipping an AI feature, evals are how you find out that last week's prompt change broke a case you had stopped checking.
Prompts for the tools this applies to
Related terms
- SWE-bench
SWE-bench is a benchmark that tests whether AI systems can resolve real GitHub issues from open-source Python repositories.
- AI code review
AI code review is using a model to examine a diff or a file for bugs, security problems, and quality issues before a human reviews it.
- Test-driven development with AI
Test-driven development with AI means writing or generating the tests first, then letting an agent implement until they pass.