Inference (AI)
Running a trained AI model on new input to get an output, such as an answer, a label or an embedding.
Last updated
What it is.
Training is when a model learns from data. Inference is every time the finished model is used: a chat reply, a document classified, an embedding created. With hosted APIs, you pay per inference call, usually by the token. If you host a model yourself, you pay for the machines that run it.
Inference cost depends on the size of the model, how many tokens go in and out, and how fast the answer must arrive. Reasoning models can produce many extra tokens before the final answer, which adds to the bill.
Why it matters
For most products, inference is the AI cost that keeps growing with usage, long after any training or fine-tuning is done. It is the number to model before launch, and the one that model routing, caching and batching are designed to bring down.
Common mistakes.
- 01
Budgeting only for training or setup. For API-based products, inference is usually the main AI cost.
- 02
Ignoring latency. A cheaper model that is too slow for the user can still be the wrong choice.
- 03
Forgetting retries and failed calls, which also count as usage.
Work it out with a tool.
Related terms.
- Token (LLM)The unit an AI language model reads and writes, and the unit most LLM APIs bill by.
- Model routingSending each AI request to the model best suited to it, instead of using one model for everything.
- Batch APIA way to send many LLM requests to run later, usually at a discount.
- LLM (large language model)An AI model trained on very large amounts of text that predicts the next piece of text, which lets it read, write and reason in language.
Guides and pages
Browse the full glossary or read answers to common questions.
