Paged attention
Managing the KV cache in fixed-size blocks rather than one contiguous allocation, so memory is not fragmented and sequences can share a batch.
By Lattice editorial · Entry last reviewed · method
What it implies
This is the change that made GPU serving practical at scale, and the reason vLLM became the default. Before it, a batch was sized for the worst-case sequence in it. After it, you size for the average and grow dynamically. If you are choosing a runtime and do not know any other criterion, this is the one to know.