What the KV cache is and how it sets the memory cost of long prompts
The KV cache grows linearly with prompt length and can exceed the model’s own weights at long context. Here is how GQA, quantization, and paged serving cut it.
What the KV cache is and how it sets the memory cost of long prompts Read Post ยป
Uncategorized