Batch API
Also called batch processing, async batch.
A way to send many LLM requests to run later, usually at a discount.
Last updated
What it is.
With a batch API you upload a file of requests, the provider runs them when it has spare capacity, and you collect the results when they finish. The provider sets a window for completion, often measured in hours rather than seconds.
Providers usually price batch calls below real-time calls. Not every feature is available in batch mode on every provider, so check the docs for the model you want.
Why it matters
A lot of AI work does not need an instant answer: tagging records, summarising documents, generating descriptions or running evals. Moving that work to batch can cut its cost without changing the model or the prompt.
Common mistakes.
- 01
Using it for anything a user is waiting on. Batch jobs can take hours.
- 02
Not handling partial failures. Some requests in a batch can fail and need resending.
- 03
Forgetting that batch and prompt caching discounts may or may not stack, depending on the provider.
Work it out with a tool.
Related terms.
- Token (LLM)The unit an AI language model reads and writes, and the unit most LLM APIs bill by.
- Prompt cachingReusing the processed start of a prompt across calls, so repeated input is billed at a lower rate.
- Inference (AI)Running a trained AI model on new input to get an output, such as an answer, a label or an embedding.
- LLM evals (evaluations)Repeatable tests that score how well a model, prompt or AI feature does a specific task.
Guides and pages
Browse the full glossary or read answers to common questions.
