51 terms

The words, defined once.

Short enough to answer a question, opinionated enough to be useful. Each entry says what the term implies for a decision, not just what it names — and every term mentioned in the essays links here automatically.

Paged attention: Managing the KV cache in fixed-size blocks rather than one contiguous allocation, so memory is not fragmented and sequences can share a batch.
KV cache: The saved attention keys and values from tokens already processed, so the model does not recompute them for every new token.
Continuous batching: Adding new sequences to a batch as others finish, rather than waiting for the whole batch to complete.
Prefix caching: Reusing the KV cache for a prompt prefix that has already been processed, instead of recomputing it.
Quantisation: Storing and computing model weights in fewer bits than they were trained in, to trade accuracy for memory and speed.
Tensor parallelism: Splitting a single layer's computation across several GPUs, so every token passes through all of them.
Speculative decoding: Using a small draft model to propose several tokens at once, which a large model then verifies in a single pass.
Time to first token: The delay between sending a request and receiving the first token of the response.
Throughput and latency: Throughput is tokens per second across the whole deployment; latency is how long one request takes. They trade directly against each other.
Constrained decoding: Masking the token space during generation so the output can only be a valid instance of a schema, grammar or regex.
Validation and repair: Generating freely, then validating the output against a schema and retrying with the error fed back.
Model router: A layer that chooses which model answers a given request, from signals known before the model is called.
The escalation trap: Routing cheap-first and retrying on a bigger model when the answer looks bad, which usually costs more than routing correctly up front.
Semantic cache: Reusing a previous response when a new request is sufficiently similar in meaning, skipping the model call entirely.
Circuit breaker: A per-provider, per-model gate that stops sending traffic somewhere after repeated failures, then probes for recovery.
Provider prompt caching: A discount providers apply when a request repeats a prefix they have already processed.
Embedding: A vector representation of text such that semantically similar inputs land near each other.
Chunking: Splitting documents into retrievable pieces, which is where RAG quality is mostly won and lost.
Hybrid search: Combining lexical retrieval (BM25) with vector retrieval, so exact matches and semantic matches are both found.
Reranking: Re-scoring retrieved candidates with a slower, more accurate model before the generator sees them.
Reciprocal rank fusion: Merging ranked lists from several retrievers by summing the reciprocal of each result's rank, rather than comparing scores.
Vector index: An approximate structure — usually HNSW or an IVF variant — that finds nearest neighbours without scanning every vector.
Recall@k: Of the documents that should answer a query, the fraction that appear in the top k retrieved.
Context window: The number of tokens a model can consider at once — a ceiling on what you can send, not a target for what you should.
LoRA: Training small low-rank matrices alongside frozen weights, so you can adapt a large model without touching it.
Supervised fine-tuning: Training on curated input/output pairs so the model adopts a specific task or response style.
Direct preference optimisation: Aligning a model to human preferences without an explicit reward model, by optimising on chosen-versus-rejected pairs.
RLHF: Reinforcement learning from human feedback: training a reward model from preferences, then optimising the policy against it.
GRPO: Group relative policy optimisation — a reinforcement-learning method that compares a group of sampled answers rather than learning a separate reward model.
Distillation: Training a smaller model against the outputs of a larger one, to get most of the behaviour at a fraction of the cost.
ZeRO: Sharding optimiser states, gradients and parameters across data-parallel workers to fit a larger model on the same GPUs.
Tool calling: A model emitting a structured request to call a function, which your code executes and returns the result of.
Agent loop: The control structure where a model decides an action, it is executed, and the result feeds the next turn until the task is done.
Model Context Protocol: An open protocol for exposing tools, resources and prompts to models, so integrations are written once rather than per model provider.
Budget ceiling: A hard limit on turns, tokens or wall-clock time, enforced by the host rather than requested in the prompt.
Durable execution: A workflow engine that checkpoints progress so work resumes after a crash, rather than restarting from nothing.
Idempotency key: A client-supplied identifier that lets a server recognise a repeated request and return the original result rather than acting twice.
Prompt injection: An attacker getting instructions into your context that your model then follows as if they came from you.
Guardrail: Any control that keeps a model from doing something you have decided it must not do.
Safety classifier: A model, distinct from the one you are serving, that labels content as safe or unsafe.
PII redaction: Detecting and masking personal data before it enters a system you do not control.
Few-shot prompting: Including worked examples in the prompt instead of only describing what you want.
System prompt: The instruction block that frames a conversation, separate from the user's messages.
Golden set: A small, curated set of real cases, with expected behaviour written down, used to tell whether a change helped.
LLM-as-judge: Using a model to score output against a written rubric, in place of a human.
Regression gate: A check in CI that blocks a merge when a change makes a scored dimension worse.
Shadow traffic: Replaying a sample of production requests against a candidate model without serving the answers.
Cost per successful outcome: Total spend divided by the count of requests that passed whatever quality bar you can define.
Open weight vs open source: Open weights means you can download and run the model. Open source additionally means the training data, training code and licence permit modification and redistribution.
Mixture of experts: A model with many parameter blocks, of which only a small number are activated per token.
Hallucination: Fluent, confident output that is not supported by the input or by evidence.