Look at the prompts your AI product actually sends. In many production systems, the first several thousand tokens are identical on every call: the system prompt, safety and style rules, tool definitions, perhaps a policy document or schema. Only the last few hundred tokens are specific to the user and the request. Yet many teams pay full input price, and wait for full processing, on that repeated prefix every time.
Prompt caching exists to fix exactly that. It is usually the cheapest optimisation available, and the most commonly wasted.
How it works
When a model processes a prompt, it builds internal state for every token. If the next request begins with exactly the same tokens, that work can in principle be reused rather than repeated. Self-hosted serving engines do this with prefix caching on top of memory management techniques such as PagedAttention5, which I cover in LLM inference serving. Hosted model providers expose the same idea as a pricing and latency feature.
The key word is prefix. Caches match from the start of the prompt. Change one token near the beginning and everything after it is a miss.
How the major providers differ
| Provider | How it's enabled | What the documentation describes |
|---|---|---|
| Anthropic (Claude) | Explicit cache breakpoints in the request | A default cache lifetime of five minutes, refreshed on use, with a one-hour option; cache writes cost more than base input and cache reads cost a small fraction of it14 |
| OpenAI | Automatic for long enough prompts | Applies automatically to prompts of 1,024 tokens or more, with hits in 128-token increments and cached tokens reported in the usage data2 |
| Google Gemini | Implicit by default on recent models, or explicit caches | Implicit caching passes savings through on hits without a guarantee; explicit caching lets you create a cache with a configurable time to live3 |
Prices, minimum sizes and lifetimes change, and differ by model and by cloud platform. Read the current documentation for the models you actually use, and measure, rather than relying on any table, including this one.
Explicit caching rewards teams who design for it. Automatic caching rewards teams who happen to have stable prefixes. Either way, prompt structure decides the outcome.
Why most teams miss
The usual culprits are small and easy to fix once you look:
- A timestamp or request ID at the top of the system prompt, so every prefix is unique.
- The user's name or account details injected into the system prompt rather than the user turn.
- Tool definitions in a different order on each request, because they're built from an unordered map.
- Retrieved documents placed before the static instructions.
- Prompt templates edited constantly, invalidating caches across all traffic with every deployment.
- Traffic too sparse for the cache lifetime: if a given prefix is used once an hour and the cache lasts minutes, it will rarely hit.
The playbook
- Order prompts from most stable to least stable. Static instructions and tool definitions first, then slowly changing reference material, then per-user context, then the request itself.
- Make serialisation deterministic. Sort tool definitions and keys, fix whitespace and formatting, and version templates so the same version always produces the same bytes.
- Move volatile values to the end. Dates, user details and request metadata belong in the final turn, not the system prompt.
- Use explicit breakpoints where available, placing them after the large stable sections. Consider the longer lifetime for traffic that is regular but not constant, and compare the extra write cost with the expected reads1.
- Measure hit rates from usage data. Providers report cached tokens in responses123. Record them per route in your gateway alongside cost, as in FOCUS and AI spend.
- Treat prompt changes as cache events. A template change resets caching for that route. Batch changes and watch cost after deployment.
- Route consistently. If you use several models or regions, send requests with the same prefix to the same destination where possible, so caches have a chance to warm. The same logic applies to replicas in self-hosted serving.
Where it pays most
- Agents. Long system prompts and large tool catalogues, repeated across many steps of the same task. Caching often makes multi-step agents economically viable; see evaluating AI agents for keeping them reliable too.
- Document question answering. When users ask several questions about the same long document, cache the document once and vary the question, which pairs well with the approach in long context or RAG.
- Batch processing with a shared instruction block and many short inputs.
- Coding assistants that resend large amounts of repository context on each turn.
It pays least for short prompts, highly personalised prompts and very low traffic, where little is shared or the cache expires between uses.
Caching, routing and the gateway
Prompt caching interacts with the other cost levers. Model routing that sends similar requests to different models can split traffic and lower hit rates on each. Retrieval that returns results in a different order each time defeats caching of the retrieved section. A gateway that owns prompt assembly, routing and measurement in one place is the simplest way to manage these trade-offs, and it's the same gateway that enforces security policies in securing MCP servers.
Data protection note
Caching keeps processed prompt state on the provider's side for a period. For regulated data, check each provider's documentation and your agreements on how cached content is isolated and retained, and whether that is compatible with obligations such as those in the HIPAA Security Rule proposal or Gulf residency rules. For the most sensitive workloads, self-hosted serving with prefix caching keeps the cache inside your own boundary.
How I can help
I review AI products for cost and latency, from prompt structure and caching to routing and serving, and build the gateway measurements that keep savings visible. See my AI architecture work, or analyse your challenge and bring the result to a call.
- 1Prompt cachingAnthropic documentation
- 2Prompt caching 201OpenAI Cookbook
- 3Context cachingGoogle AI for Developers
- 4Prompt caching for Claude modelsGoogle Cloud Vertex AI documentation
- 5Efficient Memory Management for Large Language Model Serving with PagedAttentionKwon et al. (arXiv; SOSP 2023) · 2023