If you build production applications with large language models, you quickly hit an expensive pattern: most of the tokens you send are identical from one request to the next. System instructions, tool schemas, few-shot examples, and long reference documents get reprocessed thousands of times, raising your API bill and delaying time to first token.
Understanding what is prompt caching—and shaping payloads to use it—is one of the highest-leverage AI API optimizations. Prompt caching (also called context caching or prefix caching) stores the model’s internal state for a shared prompt prefix and reuses it on later requests so the GPU does not recompute that prefix from scratch.
This guide covers how KV prefix caching works, why matching must be exact, how Anthropic, OpenAI, and Google Gemini implement and price it as of November 5, 2026 per their official docs, and how to apply the stable-prefix + varying-suffix pattern.
How Prompt Caching Works
Transformer inference has a prefill phase and a decoding phase. Before the model emits its first output token—see what a token is in AI if you need the billing unit refresher—it must read every input token and compute self-attention across every layer.
During prefill, the model builds Key (K) and Value (V) tensors for each input token. Those activations form the KV cache. Computing them for tens of thousands of tokens is expensive, which is why long prompts raise time to first token (TTFT) even when the reply is short.
Without caching, the server discards those KV tensors when the request ends. A follow-up call with the same 25,000-token system prompt recomputes everything.
Prompt caching keeps the precomputed KV state of a prompt prefix in temporary memory. On the next call, the provider checks whether the incoming prompt starts with an active cached prefix:
- Cache miss / write: No match → full prefill, store KV up to a breakpoint, assign a TTL. Writing may cost a premium over base input, depending on provider and mode.
- Cache hit / read: Exact prefix match → skip prefill for cached tokens, load saved KV, and compute attention only for the new suffix.
Prompt caching is not semantic caching or response memoization. The model still generates fresh tokens; it just skips recalculating an unchanged prefix. Providers store derived KV tensors (not raw plaintext) and isolate caches to your organization or project.
Why Prefix Matching Is Strict
Causal attention is directional: token N’s Keys and Values depend on tokens 1…N−1. Change one early character, JSON key order, or a timestamp on line one, and every later KV state is invalid.
For a hit, everything up to the breakpoint must be identical:
- Tool and function definitions (names, schemas, array order)
- System instructions and static persona text
- Few-shot examples and reference documents in the same sequence
- Model IDs and settings that change tokenization or attention
Every cached prefix also has a TTL. Hits often refresh lifetime (provider-specific); idle prefixes expire.
Provider Comparison (As of November 5, 2026)
Anthropic, OpenAI, and Gemini all support prompt caching, with different triggers, minimum lengths, TTLs, and write/read multipliers. Figures below match each provider’s official documentation as of November 5, 2026. For base rates across vendors, see our AI API pricing comparison.
| Feature | Anthropic (Claude API) | OpenAI (GPT-5.6 & Later) | Google Gemini API |
|---|---|---|---|
| Activation | Automatic (top-level cache_control) or explicit (per-block, up to 4 breakpoints) |
On by default; implicit & explicit (prompt_cache_options.mode, prompt_cache_breakpoint) |
Implicit on by default (Gemini 2.5+); explicit only on pre-Gemini 4 |
| Min length | 512 tokens on most newer models | 1,024 visible input tokens | Implicit: 4,096 (Gemini 4 Argon / 3.x Flash / 3.1 Pro Preview); 2,048 (Gemini 2.5 Flash/Pro) |
| TTL | 5m default; optional 1h | 30m default; optional "12h" in Responses API |
Implicit: short automatic window; Explicit (pre-Gemini 4): configurable (~1h default) |
| Write cost | 1.25× base (5m) or 2× (1h) | 1.25× uncached (30m) or 2.5× (12h) | Implicit: no extra write; Explicit: standard input + TTL storage |
| Read cost | 0.1× most models; 0.05× Opus 5.5 & Sonnet 5.5; 0.025× Fable 5.1 & Mythos 5.1 | 0.1× most; 0.05× GPT-6.1 Sol & GPT-6.1 Terra | Discounted cached rate on hits (often ~90% off on 2.5+ per pricing docs) |
| Prewarm | max_tokens: 0 |
prompt_cache_options.prewarm: true |
Manual explicit cache (pre-Gemini 4 only) |
Anthropic (Claude API)
Claude builds prefixes in order: tools → system → messages.
- Automatic: top-level
cache_control: {"type": "ephemeral"}places the breakpoint on the last cacheable block (handy for growing multi-turn history). Managed Agents also default to a 5-minute cache. - Explicit: attach
cache_controlto up to 4 content blocks (tools, system text, documents, turns).
Minimum cacheable length is 512 tokens on most newer models. Default TTL is 5 minutes; use {"type":"ephemeral","ttl":"1h"} for one hour. Anthropic measures lifetime from when the write/read request starts, so long streaming counts against the window.
Pricing multipliers of base input (Nov 5, 2026): 5m write 1.25×, 1h write 2×, read 0.1× (deeper on newer tiers as in the table). Absolute examples: Claude Sonnet 5.5 $2.00 base input | $2.50 5m write | $4.00 1h write | $0.10 hit | $10.00 output; Claude Opus 5.5 $4.00 | $5.00 | $8.00 | $0.20 hit | $20.00 output. Usage reports cache_creation_input_tokens, cache_read_input_tokens, and uncached input_tokens.
OpenAI API
On GPT-5.6 and later: minimum 1,024 visible input tokens; default 30m TTL with 1.25× write; Responses API optional "12h" at 2.5× write; reads 0.1× ( 0.05× on GPT-6.1 Sol / Terra). Explicit controls: prompt_cache_options.mode, prompt_cache_breakpoint, and prewarm: true. Earlier models use mainly automatic implicit caching with historically free writes and shorter inactivity windows—confirm your model tier before budgeting.
Google Gemini API
- Implicit: default on Gemini 2.5+; no code changes. Min thresholds: 4,096 on Gemini 4 Argon / 3.x Flash / 3.1 Pro Preview; 2,048 on Gemini 2.5 Flash/Pro. Put large shared context first and reuse matching prefixes quickly.
- Gemini 4 Argon+: implicit only—no explicit caching. In multi-turn chat, append about 2,050+ new tokens if you want the full prior request cached; smaller increments recompute roughly the last 2,050 tokens of the prior request at standard rates.
- Explicit (pre-Gemini 4 only): create a cache resource with custom TTL (~1h default); creation bills standard input plus storage; hits get a large discount (commonly ~90% off on 2.5+ per Google’s pricing page).
Practical Pattern: Stable Prefix + Varying Suffix
Put everything that stays constant at the front; put per-user or per-turn content at the end:
- Static
tools(fixed order) - Static system / reference corpus (no live timestamps at the top)
- Append-only conversation history—see Claude API message threads
- Varying suffix: current question, retrieval snippets, timestamps—keeping untrusted user text out of the privileged prefix also helps against prompt injection
Anthropic example (and OpenAI / Gemini notes)
{
"model": "claude-sonnet-5-5",
"max_tokens": 1024,
"tools": [
{
"name": "lookup_policy",
"description": "Search the static compliance handbook.",
"input_schema": {
"type": "object",
"properties": { "section": { "type": "string" } },
"required": ["section"]
}
}
],
"system": [
{
"type": "text",
"text": "You are a compliance assistant. Answer strictly from the handbook below.\n\n<handbook>\n[... 8,000 tokens of static policy text ...]\n</handbook>",
"cache_control": { "type": "ephemeral", "ttl": "5m" }
}
],
"messages": [
{
"role": "user",
"content": "Timestamp: 2026-11-05T14:22:10Z. Question: What is the refund window for annual enterprise plans?"
}
]
}First call writes tools + system under cache_creation_input_tokens (1.25× for 5m TTL; $2.50/MTok on Sonnet 5.5). Later identical prefixes within 5 minutes report under cache_read_input_tokens at $0.10/MTok, while only the short user suffix bills as standard input. Or use top-level automatic cache_control so Claude moves the breakpoint each turn.
OpenAI (GPT-5.6+): drop cache_control; keep static tools/instructions first and the user message last. Once the shared prefix is ≥1,024 visible tokens, default caching applies (or use explicit prompt_cache_* options / 12h TTL).
Gemini: drop cache_control. On Argon, put ≥4,096 static tokens first and keep multi-turn increments above ~2,050 when you need full prior-turn caching. On Gemini 2.5 you can also create an explicit cache.
When Caching Is Profitable
Anthropic and OpenAI GPT-5.6+ charge a write premium, so you need enough hits inside the TTL to beat uncached spend.
Example: 100,000-token prefix on Claude Sonnet 5.5 ($2.00 / $2.50 5m write / $0.10 read per MTok):
- Uncached: $0.20 per call ($2.00 over 10 calls)
- 5m cache: write $0.25, each hit $0.01 → by call 2 already cheaper; 10 calls ≈ $0.34 (~83% prefix savings)
- 1h cache (2× write): break-even around call 2–3
Sparse traffic (one call every two hours) can mean all misses and higher cost. Monitor cache_read_* vs cache_creation_* (and OpenAI/Gemini cached counters) and match TTL to real cadence.
FAQ
Does prompt caching reduce output token costs?
No. It only discounts input tokens in the matched prefix and usually lowers TTFT. Output tokens still bill at the normal output rate.
Why am I missing the cache when my system prompt looks the same?
Common causes: timestamps/UUIDs near the start, non-deterministic tools JSON key order, or prefixes below the minimum (512 on most newer Claude models, 1,024 visible on GPT-5.6+, 4,096 on Gemini 4 Argon).
Does Gemini 4 Argon support explicit context caching?
No. As of November 5, 2026, Gemini 4 Argon and later support implicit caching only. Put at least 4,096 static tokens at the front.
Are caches shared across organizations?
No. Caches are isolated per organization/project and store KV tensors, not raw prompt text.
How do I prewarm a cache?
Claude: send the cacheable prefix with max_tokens: 0. OpenAI GPT-5.6+: prompt_cache_options.prewarm: true.
Does a hit reset TTL?
Often yes for ephemeral/implicit caches, but Anthropic counts TTL from request start, so long generation eats remaining lifetime.
Sources
- Anthropic Claude API Documentation: Prompt Caching (accessed November 5, 2026)
- OpenAI API Documentation: Prompt Caching Guide (accessed November 5, 2026)
- Google Gemini API Documentation: Context Caching (accessed November 5, 2026)
- Google Gemini API Documentation: Generate Content Caching (accessed November 5, 2026)
- Google Gemini API Documentation: Pricing (accessed November 5, 2026)
