Quantisation
Storing and computing model weights in fewer bits than they were trained in, to trade accuracy for memory and speed.
By Lattice editorial · Entry last reviewed · method
What it implies
8-bit is nearly free in quality; 4-bit usually is not, and the method matters more than the bit count — AWQ and GPTQ are perceptually better than naive rounding at the same width. FP8 is the easier win on recent NVIDIA hardware. Quantisation changes your throughput numbers but not your architectural decisions, so do it after you have chosen a runtime, not before.