Context window
The number of tokens a model can consider at once — a ceiling on what you can send, not a target for what you should.
By Lattice editorial · Entry last reviewed · method
What it implies
Long contexts degrade silently rather than gracefully. Models reliably answer from the beginning and end of a long document and perform near chance on the middle. A 50k-token context is therefore a few thousand tokens of reliable signal plus a large quantity that is present, billed, and largely ignored.
Related terms
- ChunkingSplitting documents into retrievable pieces, which is where RAG quality is mostly won and lost.
- RerankingRe-scoring retrieved candidates with a slower, more accurate model before the generator sees them.
- KV cacheThe saved attention keys and values from tokens already processed, so the model does not recompute them for every new token.