{"head":{"title":"What Are LLM Evals? AI Evaluation Definition | OceanLab","description":"LLM evals are repeatable tests that score how well an AI model or feature does a task. Types of evals, how to build a useful set, and common mistakes.","canonical":"https://oceanlab.in/glossary/llm-evals"},"crumbs":[{"name":"Home","path":"/"},{"name":"Glossary","path":"/glossary"},{"name":"LLM evals","path":"/glossary/llm-evals"}],"slug":"llm-evals","term":"LLM evals (evaluations)","aka":["evals","model evaluation","AI evals"],"category":"AI building blocks","short":"Repeatable tests that score how well a model, prompt or AI feature does a specific task.","body":[["An eval is a set of test cases, each with an input and a way to judge the output. The judge can be exact checks (does the JSON have the right total?), code (does the generated code pass the tests?), a rubric scored by people, or another model grading against a rubric."],["Public benchmarks compare models in general. Product evals test your task with your data, and they are the ones that tell you whether a change is safe to ship."]],"why":["Without evals, every prompt edit, model upgrade or ",{"href":"/glossary/model-routing","text":"model routing"}," change is a guess. With them you can compare options on the same cases, catch regressions before users do and show reviewers or auditors how quality is measured."],"example":["Before moving a summarisation feature to a cheaper model, a team runs both models on two hundred real, anonymised documents. A rubric checks that each summary keeps the key dates and amounts. The cheaper model passes on most types but misses amounts in tables, so tables stay on the larger model."],"inBuild":null,"mistakes":[["Testing only easy, happy-path cases. Include the hard and messy ones."],["Letting a model grade without checking the grader against human judgments."],["Building evals once and never adding the failures found in production."]],"related":[{"href":"/glossary/model-routing","label":"Model routing","detail":"Sending each AI request to the model best suited to it, instead of using one model for everything."},{"href":"/glossary/hallucination","label":"Hallucination (AI)","detail":"When an AI model produces information that is false or made up, presented as if it were true."},{"href":"/glossary/fine-tuning","label":"Fine-tuning","detail":"Training an existing model further on your own examples so it behaves the way you want, such as following a format, tone or task more reliably."},{"href":"/glossary/guardrails","label":"Guardrails (AI)","detail":"Checks placed around an AI model that block, fix or flag unsafe, off-topic or incorrect inputs and outputs."}],"tools":[{"href":"/tools/llm-model-picker","label":"LLM model picker","detail":"Free tool"}],"links":[{"href":"/ai-labs/evals","label":"Evals and benchmarks","detail":"OceanLab Labs builds evals for AI labs."},{"href":"/ai","label":"AI by industry","detail":"AI use cases, costs and risks per industry."},{"href":"/ai-automation","label":"AI automation","detail":"How OceanLab automates company workflows with AI."}],"sources":[],"legal":false,"updatedAt":"2026-10-25"}