{"head":{"title":"What Is a Batch API for LLMs? Definition | OceanLab","description":"A batch API lets you send many LLM requests to run later, usually at a discount. How it works, which jobs suit it, an example and the mistakes to avoid.","canonical":"https://oceanlab.in/glossary/batch-api"},"crumbs":[{"name":"Home","path":"/"},{"name":"Glossary","path":"/glossary"},{"name":"batch API","path":"/glossary/batch-api"}],"slug":"batch-api","term":"Batch API","aka":["batch processing","async batch"],"category":"LLM costs","short":"A way to send many LLM requests to run later, usually at a discount.","body":[["With a batch API you upload a file of requests, the provider runs them when it has spare capacity, and you collect the results when they finish. The provider sets a window for completion, often measured in hours rather than seconds."],["Providers usually price batch calls below real-time calls. Not every feature is available in batch mode on every provider, so check the docs for the model you want."]],"why":["A lot of AI work does not need an instant answer: tagging records, summarising documents, generating descriptions or running ",{"href":"/glossary/llm-evals","text":"evals"},". Moving that work to batch can cut its cost without changing the model or the prompt."],"example":["An insurer wants every claim note from the past year classified by claim type. Nobody is waiting on any single result, so the team sends the notes as batch jobs overnight and loads the results into their reporting database in the morning."],"inBuild":null,"mistakes":[["Using it for anything a user is waiting on. Batch jobs can take hours."],["Not handling partial failures. Some requests in a batch can fail and need resending."],["Forgetting that batch and ",{"href":"/glossary/prompt-caching","text":"prompt caching"}," discounts may or may not stack, depending on the provider."]],"related":[{"href":"/glossary/token","label":"Token (LLM)","detail":"The unit an AI language model reads and writes, and the unit most LLM APIs bill by."},{"href":"/glossary/prompt-caching","label":"Prompt caching","detail":"Reusing the processed start of a prompt across calls, so repeated input is billed at a lower rate."},{"href":"/glossary/inference","label":"Inference (AI)","detail":"Running a trained AI model on new input to get an output, such as an answer, a label or an embedding."},{"href":"/glossary/llm-evals","label":"LLM evals (evaluations)","detail":"Repeatable tests that score how well a model, prompt or AI feature does a specific task."}],"tools":[{"href":"/tools/llm-api-cost-calculator","label":"LLM API cost calculator","detail":"Free tool"}],"links":[{"href":"/pricing/categories/llm-api","label":"LLM API pricing","detail":"Model prices side by side, with sources."},{"href":"/pricing","label":"Pricing library","detail":"Software and API prices, each with its source and date."}],"sources":[],"legal":false,"updatedAt":"2026-10-25"}