Continuous batching
Adding new sequences to a batch as others finish, rather than waiting for the whole batch to complete.
By Lattice editorial · Entry last reviewed · method
What it implies
It exploits the fact that sequence lengths vary wildly within a batch — waiting for the longest wastes most of the slot. Under steady, low-concurrency traffic there is never enough in flight to fill a batch, so this buys you nothing. That traffic-shape dependency is the biggest single factor in picking a runtime.