Prompt caching
Also called context caching.
Reusing the processed start of a prompt across calls, so repeated input is billed at a lower rate.
Last updated
What it is.
When many requests start with the same long prefix, such as instructions, tool definitions or a reference document, the provider can keep the processed prefix for a short time. Later calls that start with exactly the same prefix read it from the cache instead of processing it again.
Cached reads are billed below normal input tokens. Writing to the cache can cost more than normal input on some providers, and the cache expires after a period without use. Some providers cache automatically; others need you to mark where the cached part ends.
Why it matters
For chat assistants, agents and document tools, most input is the same on every call. Caching can make that repeated part much cheaper and often faster to start answering. Whether it pays off depends on how often the same prefix is reused before it expires.
Common mistakes.
- 01
Putting changing content, like a timestamp or user name, at the very start. Any change in the prefix breaks the cache.
- 02
Caching prompts that are rarely reused before the cache expires, so you pay the write cost without the savings.
- 03
Comparing providers on the list input price alone when most of your input would be cached.
Work it out with a tool.
Related terms.
- Token (LLM)The unit an AI language model reads and writes, and the unit most LLM APIs bill by.
- Context windowThe most tokens a model can take in and produce in one request.
- Batch APIA way to send many LLM requests to run later, usually at a discount.
- Model routingSending each AI request to the model best suited to it, instead of using one model for everything.
Guides and pages
Browse the full glossary or read answers to common questions.
