LLM evals (evaluations)
Also called evals, model evaluation, AI evals.
Repeatable tests that score how well a model, prompt or AI feature does a specific task.
Last updated
What it is.
An eval is a set of test cases, each with an input and a way to judge the output. The judge can be exact checks (does the JSON have the right total?), code (does the generated code pass the tests?), a rubric scored by people, or another model grading against a rubric.
Public benchmarks compare models in general. Product evals test your task with your data, and they are the ones that tell you whether a change is safe to ship.
Why it matters
Without evals, every prompt edit, model upgrade or model routing change is a guess. With them you can compare options on the same cases, catch regressions before users do and show reviewers or auditors how quality is measured.
Common mistakes.
- 01
Testing only easy, happy-path cases. Include the hard and messy ones.
- 02
Letting a model grade without checking the grader against human judgments.
- 03
Building evals once and never adding the failures found in production.
Work it out with a tool.
Related terms.
- Model routingSending each AI request to the model best suited to it, instead of using one model for everything.
- Hallucination (AI)When an AI model produces information that is false or made up, presented as if it were true.
- Fine-tuningTraining an existing model further on your own examples so it behaves the way you want, such as following a format, tone or task more reliably.
- Guardrails (AI)Checks placed around an AI model that block, fix or flag unsafe, off-topic or incorrect inputs and outputs.
Guides and pages
Browse the full glossary or read answers to common questions.
