Speculative decoding
Using a small draft model to propose several tokens at once, which a large model then verifies in a single pass.
By Lattice editorial · Entry last reviewed · method
What it implies
It is exact — the output distribution is unchanged — and speeds up decoding where it is bottlenecked on memory bandwidth rather than compute. The gain depends on the draft model agreeing with the target, so it helps most for predictable text and least for genuinely creative work. Worth trying when you have already exhausted batching and quantisation.