Throughput and latency
Throughput is tokens per second across the whole deployment; latency is how long one request takes. They trade directly against each other.
By Lattice editorial · Entry last reviewed · method
What it implies
Batching raises throughput by making each request slower, so the two cannot be optimised independently. What you want depends entirely on traffic shape: an interactive product needs bounded latency under whatever concurrency arrives, a batch job wants total tokens cheap. Benchmark on a concurrency sweep and plot both — never single-stream latency, which will point you at the wrong runtime.