KV cache
The saved attention keys and values from tokens already processed, so the model does not recompute them for every new token.
By Lattice editorial · Entry last reviewed · method
What it implies
It is what makes generation incremental rather than quadratic, and its size is the main thing that determines how many concurrent sequences fit on a GPU. It grows with context length and batch size, which is why long contexts and high concurrency compete for the same memory.
Related terms
- Paged attentionManaging the KV cache in fixed-size blocks rather than one contiguous allocation, so memory is not fragmented and sequences can share a batch.
- Prefix cachingReusing the KV cache for a prompt prefix that has already been processed, instead of recomputing it.
- Context windowThe number of tokens a model can consider at once — a ceiling on what you can send, not a target for what you should.