When you run a large language model on a long document, the heaviest part of the memory bill is often not the model’s weights. It is a cache the model keeps while it reads the prompt. This cache is called the KV cache, and it is what turns prompt length into a memory decision.
The core idea is simple. A transformer builds each token from small vectors called queries, keys, and values. As the model writes output one token at a time, it keeps looking back at everything it has already read. The KV cache stores the keys and values of the tokens it has seen, so each new step reads them instead of rebuilding them.
What the KV cache is
Attention is the part of a transformer that lets one token weigh every other token. To do this, it uses three vectors per token: a query, a key, and a value. During generation, the model predicts the next token by attending to all previous tokens. Without any shortcut, each new token forces the model to recompute the keys and values of every earlier token. As a result, the attention work grows with the square of the sequence length, because you repeat the cost one step at a time over the whole sequence. This comparison of KV cache strategies spells out the quadratic baseline, and the FastKV paper makes the same point for the prefill stage.
The KV cache is the shortcut. The model writes the keys and values of each token into the cache as it goes, and a later step reads those stored vectors instead of rebuilding them. In other words, it turns a quadratic cost into a linear one. The characterization paper notes that reusing cached entries, rather than recomputing attention over the full prompt, is what reduces the complexity from quadratic to linear.
Think of the cache as a running summary of the prompt. It holds, for every token seen so far, the information the model needs to keep attending to that token. One reference figure makes the scale concrete. For the 13B OPT model, a single token takes 800 KB of cache, and a 2048-token request holds up to about 1.6 GB. The PagedAttention paper works through that exact example.
Why the cache grows with the prompt
Now the direct answer to the second question: the cache grows in direct proportion to the prompt. The model stores one key and one value for each token, in every layer, for each key-value head. So the cache size scales linearly with context length. Double the tokens and you double the memory. The KV cache characterization paper states that the cache grows multilinearly with sequence length, layer depth, and head width.
Four numbers in the model’s architecture set the exact size. They are the number of layers, the number of key-value heads, the head dimension, and the bytes per element. The leading factor of two covers the key tensor and the value tensor. The PagedAttention paper shows the same four factors in its 800 KB per token worked example, where the key-value head count and head dimension are folded into the 5120 hidden state.
Two architecture choices matter a lot here. First is the number of layers, because a deeper model stores one block per layer. Second is the number of key-value heads. Most recent models use grouped-query attention, which shares keys and values across several query heads. The GQA paper introduced this scheme. That single change shrinks the cache, since fewer key-value heads mean fewer vectors to store. Multi-query attention goes one step further and keeps a single shared set of heads. Both trade a little of the flexibility of full multi-head attention for a smaller cache and faster decoding. The NVIDIA attention co-design post explains that a larger head group improves decode throughput.
Prefill, then decode
Running a prompt happens in two phases, and the cache behaves differently in each. In the prefill phase the model reads the entire prompt at once and builds the cache from scratch. This step is compute bound, and its attention work scales with the square of the input length. The NVIDIA post puts it plainly: prefill latency scales quadratically with input length.
Then comes decode. The model emits one token at a time, and at every step it reads the whole existing cache. Decode is memory bound, because each step pulls the entire cache from memory, and its cost grows linearly with how long the cache already is. The NVIDIA post calls decode latency linear in the cache length, and memory bound. This is why a long prompt is a memory problem rather than a speed problem: the cache is large, and it is read again and again.
The prefill phase reads the whole prompt and builds the KV cache. Decode then appends one token at a time, and each step reads the entire cache. The cache grows linearly with the number of tokens.
How big it gets
Let’s look at four real open models, each run at batch size one, which means one prompt with no parallel requests. The table uses the values from each model’s configuration file and the formula above. Mistral and Qwen were read directly from their config files; the Llama 3.1 8B and Llama 3.3 70B values were cross-verified because those repositories are gated.
| Model | Layers | KV heads | GQA | Per token (bf16) | 8K | 32K | 128K | 256K |
|---|---|---|---|---|---|---|---|---|
| Llama 3.1 8B | 32 | 8 | 4:1 | 128 KiB | 1.0 GiB | 4.0 GiB | 16.0 GiB | 32.0 GiB |
| Llama 3.3 70B | 80 | 8 | 8:1 | 320 KiB | 2.5 GiB | 10.0 GiB | 40.0 GiB | 80.0 GiB |
| Mistral Small 24B | 40 | 8 | 4:1 | 160 KiB | 1.25 GiB | 5.0 GiB | 20.0 GiB | 40.0 GiB |
| Qwen2.5 7B | 28 | 4 | 7:1 | 56 KiB | 0.44 GiB | 1.75 GiB | 7.0 GiB | 14.0 GiB |
GQA is grouped-query attention, shown here as the ratio of query heads to key-value heads. All values are in bf16 at batch size one. Three of the four models ship a 128K window, and the 256K column is extrapolated past it. Mistral Small 24B ships a 32K window, so the 128K and 256K points on that row are extrapolations past it.
KV cache memory in GiB against context length for four models in bf16. The dashed and dotted lines mark a 24 GB GPU (22.3 GiB) and an 80 GB GPU (74.5 GiB). Three of the four models ship a 128K window, and the 256K point is the arithmetic cost of extending past it. Mistral Small 24B ships a 32K window, so the 128K and 256K points on that row are extrapolations past it.
How to read the numbers
Read the chart in two ways. First, every line is a straight line, and that is the point: the cache is proportional to length. Second, the gaps between models are large. The 70B model holds 80 GiB of cache at 256K, while the 7B model holds 14 GiB. In other words, the difference is driven mostly by layer depth and key-value head count. Three of the four models ship a 128K window, and the 256K point on each line extends past it. Mistral Small 24B ships a 32K window, so on its row the 128K and 256K points are extrapolations past its native window.
Two practical numbers fall out of this. Store the cache in FP8 or INT8 instead of bf16 and every value halves, because each element then uses one byte instead of two. The vLLM cache configuration docs list the FP8 family as supported cache data types, and the CacheDType definition in the vLLM source adds INT8 among them. And the crossover with weights is real. The 8B model’s weights are about 15 GiB in bf16, and its cache at 128K already reaches 16 GiB. At 256K the cache is double the weights. The inference survey observes that past about 50K tokens, quantizing the KV cache can significantly decrease the memory it consumes.
What you can do about it
You have three levers, and they stack. The first is the model itself. Choose one with grouped-query or multi-query attention and few key-value heads, because that directly shrinks the cache. The GQA paper and the NVIDIA post both point the same way.
The second is precision. Quantizing the cache to FP8 or INT8 halves its size with little quality loss in most deployments. vLLM exposes this as the cache data type, and it needs recent CUDA and hardware support. The vLLM cache docs document the supported values.
The third is how the cache is managed. PagedAttention, the memory manager in vLLM, keeps the cache in fixed blocks instead of one big contiguous region. That removes most of the wasted space, lets separate requests share blocks, and is what lets one server hold many long requests at once. It lifts throughput by two to four times. The PagedAttention paper reports those gains. This is the same bottleneck that AI inference routing targets, where one endpoint has to serve many concurrent requests.
In addition, prefix caching adds a fourth lever when the same long document is asked about repeatedly. The engine reuses the stored cache for the shared prefix and skips the prefill work on later requests. It speeds up the prefill phase, not the decode phase. The vLLM prefix caching docs describe this behavior. That is the same pattern behind retrieval-augmented generation, where a model reads your private documents.
If you run on constrained hardware, the total memory budget, weights plus cache, is what decides whether a model fits at all. That trade-off is the same one covered in our look at running models with Intel’s OpenVINO.
Why this matters for long enterprise documents
In practice, long documents are the normal input in many real systems. A contract, an annual report, a software manual, a clinical note, a city permit file. Each of these can run into tens of thousands of tokens, and at that length the KV cache, not the weights, is often the binding constraint on memory. The inference survey makes the point.
Ultimately, when you plan to run a model on such inputs, budget the cache the way you budget the weights. Size it from the formula, check it against your real hardware, and reach for grouped-query models, FP8 or INT8, and paged serving before you reach for a bigger GPU. That is the difference between a long-document workflow that fits on the hardware you already run, and one that needs a second machine.
This is where the KV cache meets the reason for running these models at all. If the goal is to let data and AI drive better decisions across the systems that serve society, then long enterprise documents are exactly where the value lives. The cache is what decides whether that workflow runs on the hardware you have.
References
- Hugging Face: Mistral-Small-24B-Instruct-2501 model card and config.json
- Hugging Face: Qwen2.5-7B model card and config.json
- Hugging Face: Llama-3.1-8B model card
- Hugging Face: Llama-3.3-70B-Instruct model card
- arXiv: GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- arXiv: Efficient Memory Management for Large Language Model Serving with PagedAttention
- vLLM: Automatic Prefix Caching
- vLLM: Cache configuration
- NVIDIA Technical Blog: Co-Designing AI Model Attention for Fast, Interactive Long-Context Inference
- ACL Anthology: FastKV, Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration
- arXiv: LLM Inference Unveiled, Survey and Roofline Model Insights
- arXiv: Comparative Characterization of KV Cache Management Strategies for LLM Inference
- vLLM: cache.py, the CacheDType cache data type definition
