Prefix caching
Reusing the KV cache for a prompt prefix that has already been processed, instead of recomputing it.
By Lattice editorial · Entry last reviewed · method
What it implies
If a large share of your requests begin with the same few hundred tokens — a long system prompt, a cached document, a shared tool schema — this is worth more than every other optimisation combined. If your prompts are short and unrelated, neither vLLM's automatic prefix caching nor SGLang's RadixAttention will do much. It also makes a stable system prompt dramatically cheaper than a per-request one.