ZeRO
Sharding optimiser states, gradients and parameters across data-parallel workers to fit a larger model on the same GPUs.
By Lattice editorial · Entry last reviewed · method
What it implies
The memory optimisation that makes fine-tuning large models on modest clusters possible. If your model already fits on one device, ZeRO buys nothing and costs communication.