Tensor parallelism
Splitting a single layer's computation across several GPUs, so every token passes through all of them.
By Lattice editorial · Entry last reviewed · method
What it implies
It is the default answer to 'the model does not fit' and the easiest to get wrong, because it adds a collective operation to every forward pass and the cost grows with sequence length. If your model fits on one GPU with room for KV cache, do not do it. Two GPUs running two replicas behind a load balancer will usually beat one GPU running a sharded 70B.