Mixture of experts
A model with many parameter blocks, of which only a small number are activated per token.
By Lattice editorial · Entry last reviewed · method
What it implies
It decouples parameter count from per-token compute, which is how you get a large model's quality at a small model's cost. The practical consequences are operational: the weights are large even though compute is not, and serving needs to be expert-aware.