LotookLotook
RSS feed
AI Guides · 10 min read

What Is a KV Cache? How LLMs Reuse Attention States

Learn what a KV cache is, why it speeds up LLM decoding, how it grows with context length, and how it relates to prompt caching and API cost.

By Chunyu Cai

Illustration of a sequence of colored token tiles feeding into stacked cache slabs that emit a faster glowing output stream

A KV cache (key-value cache) is an inference optimization where a large language model stores the per-layer Key and Value attention tensors computed for already processed tokens. During autoregressive decoding, the model reuses those stored states instead of recomputing every earlier token at every step, cutting redundant computation as sequences grow.

Whether you call hosted APIs or run open-weight models on your own GPUs, the KV cache sits at the center of latency, memory pressure, and throughput. Understanding how it works explains why time to first token feels different from streaming speed, why long contexts consume so much VRAM, and why prompt caching across API requests requires exact prefix matches.

How Does Attention Work Without a KV Cache?

Without a KV cache, an autoregressive Transformer must recompute the attention projections for the entire sequence from scratch every time it generates a single new token in an AI model.

In each Transformer decoder layer, self-attention projects every input token representation into three vectors:

  • Query ($Q$): Represents the current position looking for relevant context.
  • Key ($K$): Represents what each position offers for queries to match against.
  • Value ($V$): Holds the information aggregated once attention weights are calculated from the Query-Key dot products.

Because decoder-only language models generate text one token at a time, generating token $t+1$ only requires the Query vector of the latest token to attend to the Keys and Values of tokens $1$ through $t$. If the runtime discards those Keys and Values after emitting token $t$, step $t+1$ has to run every prefix token back through every layer just to recreate the exact same $K$ and $V$ tensors. As the prompt and output grow longer, that repeated work scales quadratically across the full generation pass.

Why Does Causal Attention Make KV Caching Possible?

Causal masking guarantees that a token’s Key and Value tensors depend only on the tokens that came before it, never on tokens generated later.

In a causal (left-to-right) decoder, token $i$ is blocked from attending to any position $j > i$. Because information only flows forward in the sequence:

  1. The representation of token $i$ at layer $l$ is completely fixed once tokens $1 \dots i$ have been processed.
  2. Appending token $i+1$ at the end of the sequence does not alter the hidden states, Keys, or Values of positions $1 \dots i$ in any layer.
  3. The runtime can append the newly computed $K_{i+1}$ and $V_{i+1}$ vectors to a per-layer buffer—the KV cache—and pass only the single newest token into the next decoding step.

As documented in the Hugging Face Transformers KV cache guide and the Transformers cache architecture overview, caching turns each decoding step from a full-sequence pass into an incremental single-token update that reads historical Keys and Values from memory.

How Do Prefill and Decode Use the KV Cache Differently?

LLM inference splits into two distinct phases—prefill and decode—that interact with the KV cache in fundamentally different ways and hit different hardware bottlenecks.

Inference Phase What Happens to the KV Cache Primary Hardware Bottleneck User-Visible Metric
Prefill (Prompt Processing) Computes $K$ and $V$ for all prompt tokens in parallel and populates the initial cache across every layer. Compute-bound (large matrix-matrix multiplications across the full prompt). Time to First Token (TTFT).
Decode (Token Generation) Computes $Q$, $K$, and $V$ for one new token at a time, appends the new $K$/$V$ entry, and reads the full cached history. Memory-bandwidth-bound (loading model weights and growing KV states for each step). Inter-token latency / output streaming speed.

As explained in NVIDIA’s technical explanation of prefill, decode, and KV caching, the prefill phase processes all input tokens simultaneously. Because the full prompt is known upfront, GPUs can batch the projections into dense matrix operations, saturating compute units and populating the KV cache in one pass. Long prompts—such as multi-document contexts in retrieval-augmented generation (RAG) pipelines—directly increase prefill work and push up Time to First Token.

Once the first output token is emitted, inference enters the decode phase. Each step processes just one new token per sequence, appends its Key and Value vectors to the cache, and reads all previously stored $K$ and $V$ tensors from GPU high-bandwidth memory (HBM) to compute attention. Because arithmetic intensity is low per step while memory reads scale with sequence length and batch size, decoding quickly becomes bound by memory bandwidth and available VRAM rather than raw compute FLOPs.

How Much Memory Does a KV Cache Use?

The memory footprint of a KV cache grows linearly with sequence length, batch size, layer count, and the number of Key-Value attention heads.

From the tensor shapes used in standard Transformer caches (where each layer stores one Key tensor and one Value tensor shaped across batch, KV heads, sequence length, and head dimension), the per-sequence cache size can be expressed symbolically as:

bytes ≈ 2 × layers × kv_heads × head_dim × seq_len × bytes_per_element

Every term in that formula represents a lever that model architects and inference engineers pull to keep memory under control:

  • 2: Accounts for storing both Key (K) and Value (V) tensors.
  • layers: Total number of Transformer decoder blocks in the model.
  • kv_heads and head_dim: The number of Key/Value attention heads per layer and the dimension of each head.
  • seq_len: Total cached tokens (prompt tokens plus generated tokens so far).
  • bytes_per_element: Numerical precision of the stored tensors (for example, 2 bytes for FP16/BF16, 1 byte for 8-bit formats, or smaller fractional footprints with 4-bit KV cache quantization such as NVFP4).

Shrinking kv_heads: MHA, MQA, and GQA

In classic Multi-Head Attention (MHA), every Query head has its own dedicated Key head and Value head (kv_heads = query_heads). When context windows expanded and batch sizes grew, storing separate K and V projections for every head became a major memory bottleneck.

Two architectural changes directly reduce kv_heads in the formula above:

Alongside GQA, inference engines also reduce bytes_per_element by quantizing cached K and V tensors into lower-precision formats, allowing longer contexts or larger concurrent batches to fit in GPU memory.

How Do Serving Engines Manage KV Cache Memory at Scale?

Production serving engines manage KV cache memory by splitting contiguous tensors into fixed-size blocks so GPUs can pack more concurrent requests without wasting VRAM.

In naive implementations, a serving runtime pre-allocates a contiguous chunk of GPU memory sized to each request’s maximum possible sequence length. Because actual output lengths aren’t known ahead of time, pre-allocation creates severe internal and external memory fragmentation: reserved slots sit unused while new requests are blocked from running.

Kwon et al. addressed this bottleneck with PagedAttention in Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv:2309.06180), the core mechanism behind vLLM. Inspired by operating system virtual memory and paging:

  • Non-contiguous fixed-size blocks: PagedAttention partitions each request’s KV cache into fixed-size blocks that hold Keys and Values for a set number of tokens, mapped via a logical-to-physical block table.
  • On-demand allocation: New physical blocks are allocated only as new tokens are generated, virtually eliminating internal fragmentation.
  • Block-level prefix sharing: When multiple requests share an identical prompt prefix (such as a system prompt or shared few-shot examples), their logical block tables can point to the same physical KV blocks in memory, using copy-on-write only when sequences diverge.

In the vLLM paper’s evaluation (arXiv:2309.06180), this paged KV management achieved 2–4× higher throughput than FasterTransformer and Orca at comparable latency by fitting significantly larger batch sizes into the same GPU memory. Modern serving stacks including vLLM and NVIDIA TensorRT-LLM’s KV cache reuse system rely on block-based KV management to reuse shared prefixes and maximize concurrency.

What Is the Difference Between a KV Cache and Prompt Caching?

A KV cache is the underlying per-request inference mechanism that reuses attention states across decoding steps, whereas prompt caching is a cross-request serving and API feature that persists and reuses those KV blocks across separate calls that share an identical prefix.

Because developers often encounter both terms when optimizing latency and bills, it helps to separate what happens inside a single generation call from what happens across multiple API requests:

Dimension KV Cache (Within-Request Decoding) Prompt Caching (Cross-Request Prefix Reuse)
Scope Single request, across token generation steps. Multiple requests over a short retention window.
What It Avoids Recomputing earlier tokens inside the same response as each new token is decoded. Re-running the prefill pass on repeated static prompt prefixes across separate calls.
Lifecycle Created during prefill, appended during decode, freed or evicted when the request completes. Retained in block storage (GPU or host memory) with an eviction policy and TTL for future calls.
Billing Visibility Built into standard inference; not billed as a separate line item. Often exposed by API providers as discounted cached-input token pricing and lower TTFT.

Notice why causal attention dictates a strict rule for prompt caching: prefix matching must be exact from the very first token. Because every token’s Key and Value tensors depend on the exact sequence of tokens before it, changing even a single timestamp, user ID, or instruction at the top of a prompt invalidates the cached $K$ and $V$ states for every token that follows.

For provider-specific retention rules, minimum token thresholds, and discount structures, see our full guide to what prompt caching is and how providers price it.

Practical Takeaways for Developers and Teams

Understanding how the KV cache behaves translates directly into better prompt architecture, steadier latency, and lower infrastructure spend:

  1. Keep static content at the very beginning of your prompt: Place system instructions, tool schemas, and shared reference documents before any dynamic user input or per-request metadata. An identical prefix lets both hosted APIs and local engines like vLLM or TensorRT-LLM reuse cached KV blocks instead of running redundant prefill passes.
  2. Watch how context length scales concurrency on self-hosted GPUs: Because KV cache size grows linearly with seq_len, doubling average context length cuts the number of concurrent requests your GPU memory can hold at once. When choosing open-weight models for self-hosting, favor architectures that use Grouped-Query Attention (GQA) and runtimes that support paged KV allocation and KV quantization.
  3. Trim unnecessary context even when context windows are huge: Feeding huge histories into every turn inflates prefill latency, increases KV memory pressure, and drives up token bills. Combine prefix caching with targeted retrieval and periodic summarization—core tactics covered in our guide on how to reduce AI API costs and our AI API pricing comparison.

Frequently Asked Questions

Does a KV cache change the model’s output quality?

A standard KV cache kept in the model’s native inference precision (such as FP16 or BF16) is mathematically equivalent to recomputing the prefix at every step, because causal masking prevents future tokens from altering past Key and Value vectors. Quality tradeoffs only enter the picture if you apply lossy KV cache quantization (such as 8-bit or 4-bit compression) or evict older tokens to save memory.

Why can’t a KV cache reuse tokens that appear in the middle of a prompt if the beginning changed?

In a causal Transformer, every token’s representation incorporates attention over all preceding tokens in the sequence. If the first ten tokens of a prompt change, the hidden states—and therefore the Key and Value projections—of token eleven onward will also change. Because those states depend on the exact prefix before them, serving engines can only reuse cached KV blocks across requests when the prefix matches from the start.

Does the KV cache hold the model weights?

No. Model weights are static parameters learned during training and shared across all requests served by the GPU. The KV cache holds dynamic, request-specific activation tensors (the Key and Value projections for the tokens currently in context). Total GPU memory during inference must fit the static model weights, temporary runtime activations, and the dynamic KV caches for all active requests in the batch.

Why does generating more output tokens slow down when concurrency is high?

During the decode phase, every new token step requires reading the stored Key and Value tensors for all prior tokens in every active request in the batch. As sequence lengths and concurrent request counts climb, the volume of KV data transferred from GPU memory each step increases, saturating memory bandwidth and forcing the scheduler to limit batch size or wait on memory reads.

Sources

Written by

Chunyu Cai

Chunyu Cai is the CEO of Lotook, LLC and an entrepreneur who writes about AI models, agents, and how they change everyday work. Chunyu holds a master's degree from Imperial College London.

X (@ChunyuCai)

All articles →
Illustration of documents and books flowing into a glowing funnel, then through a neural-network brain that produces a finished document
AI Guides · 10 min ·

What Is RAG? Retrieval-Augmented Generation Explained

RAG retrieves relevant documents and feeds them to a model before it answers, so responses can use current or private data instead of training memory alone.

Illustration of an AI model context window becoming cluttered with stale tokens and losing focus on core instructions
AI Guides · 8 min ·

What Is Context Rot? Why Long Prompts Degrade LLM Output

Learn what context rot is in large language models, how it differs from the lost-in-the-middle effect, and practical ways to prevent prompt degradation.

Illustration of teal, gold, and coral token stacks flowing through a funnel into a smaller path beside a compact cube
AI Guides · 12 min ·

How to Reduce AI API Costs: A Practical Developer Checklist

Learn how to reduce AI API costs: meter tokens, route to smaller models, cap outputs, trim history, and use prompt caching, batch APIs, and quota alerts.