{"head":{"title":"What Is Inference in AI? Definition and Costs | OceanLab","description":"Inference is running a trained AI model to get an output, such as an answer or a label. How it differs from training, what drives its cost, and mistakes.","canonical":"https://oceanlab.in/glossary/inference"},"crumbs":[{"name":"Home","path":"/"},{"name":"Glossary","path":"/glossary"},{"name":"inference","path":"/glossary/inference"}],"slug":"inference","term":"Inference (AI)","aka":[],"category":"LLM costs","short":"Running a trained AI model on new input to get an output, such as an answer, a label or an embedding.","body":[["Training is when a model learns from data. Inference is every time the finished model is used: a chat reply, a document classified, an ",{"href":"/glossary/embedding","text":"embedding"}," created. With hosted APIs, you pay per inference call, usually by the ",{"href":"/glossary/token","text":"token"},". If you host a model yourself, you pay for the machines that run it."],["Inference cost depends on the size of the model, how many tokens go in and out, and how fast the answer must arrive. Reasoning models can produce many extra tokens before the final answer, which adds to the bill."]],"why":["For most products, inference is the AI cost that keeps growing with usage, long after any training or ",{"href":"/glossary/fine-tuning","text":"fine-tuning"}," is done. It is the number to model before launch, and the one that ",{"href":"/glossary/model-routing","text":"model routing"},", caching and batching are designed to bring down."],"example":["A document tool runs three model calls per page: extract, check and summarise. At low volume the cost is small. When a customer uploads a large archive, inference cost grows in step with pages, so the team routes the check step to a smaller model."],"inBuild":null,"mistakes":[["Budgeting only for training or setup. For API-based products, inference is usually the main AI cost."],["Ignoring latency. A cheaper model that is too slow for the user can still be the wrong choice."],["Forgetting retries and failed calls, which also count as usage."]],"related":[{"href":"/glossary/token","label":"Token (LLM)","detail":"The unit an AI language model reads and writes, and the unit most LLM APIs bill by."},{"href":"/glossary/model-routing","label":"Model routing","detail":"Sending each AI request to the model best suited to it, instead of using one model for everything."},{"href":"/glossary/batch-api","label":"Batch API","detail":"A way to send many LLM requests to run later, usually at a discount."},{"href":"/glossary/llm","label":"LLM (large language model)","detail":"An AI model trained on very large amounts of text that predicts the next piece of text, which lets it read, write and reason in language."}],"tools":[{"href":"/tools/llm-api-cost-calculator","label":"LLM API cost calculator","detail":"Free tool"},{"href":"/tools/llm-model-picker","label":"LLM model picker","detail":"Free tool"},{"href":"/tools/ai-chatbot-cost","label":"AI chatbot cost","detail":"Free tool"}],"links":[{"href":"/pricing/categories/llm-api","label":"LLM API pricing","detail":"Model prices side by side, with sources."},{"href":"/ai","label":"AI by industry","detail":"AI use cases, costs and risks per industry."}],"sources":[],"legal":false,"updatedAt":"2026-10-25"}