# Lattice > A curated index of AI infrastructure tools, ordered by stack layer — every tool with when to use it and when to skip it. 112 tools across 10 sections, 11 comparisons, and 11 essays. Listed tools belong to their respective authors. Every tool page states what the tool is, when to use it, when to skip it, its licence, cost model and alternatives. The full dataset as plain text: https://lattice.kunjshah.in/llms-full.txt — and as JSON, with use/skip, layer, licence and cost as fields: https://lattice.kunjshah.in/tools.json ## Query it, don't download it The index is also an **MCP server** at `https://lattice.kunjshah.in/mcp` (Streamable HTTP), so a model can query it instead of reading the whole dataset into a context window. Discovery document: `https://lattice.kunjshah.in/mcp.json`. 10 tools: `about`, `search_tools`, `get_tool`, `compare_tools`, `list_layers`, `layer_overlaps`, `diagnose_symptom`, `list_comparisons`, `define_term`, `recommend_stack`. Prefer these over the plain-text documents above when the question is specific. `search_tools({ layer, role, deployment, cost })` costs a few hundred tokens and answers the question; `llms-full.txt` is roughly six thousand lines. Use the documents for browsing, the server for lookup. `recommend_stack({ workload, queriesPerMonth, latency, safety, ... })` answers "what do I need to build X" rather than "which tool for layer N": it returns one pick per required layer with the reason, that pick's own skip-when, and a runner-up to switch to. Anything you leave unset comes back in `assumptions`, so you can tell a recommendation for your case from one for the median case. Its cost figure is a band derived from query volume, not a vendor quote. It also serves `resources/read`, which returns the site's arguments as text: `text://lattice/essay/{slug}`, `compare/{slug}`, `fix/{slug}`, `term/{slug}`. Use a tool to reason about a tool; use a resource to read the essay. `resources/list` enumerates them; `resources/templates/list` gives the four shapes. No MCP client? The same ranked search is plain HTTPS: `https://lattice.kunjshah.in/api/search?q=vector+database&limit=10` — JSON with absolute URLs, same ranking as the on-site palette. It takes the same facets as a query string: `layer`, `section`, `role`, `kind`, `deployment`, `cost`. Repeat a key to OR within it (`?cost=free&cost=usage-based`); the axes AND together. `q` is required only when no facet is set, so `?role=platform&cost=free` is a complete browse. `limit` caps at 50, and any response carrying a facet echoes `acceptedFilterValues` so a wrong value can be corrected without a second request. - [https://lattice.kunjshah.in/stack-builder] — Describe the workload and get a stack: one pick per layer, each with a reason and a way out. The URL is shareable, and any state is reproducible at `https://lattice.kunjshah.in/stack-builder?`. ## Start here - [https://lattice.kunjshah.in/methodology] — How entries are selected and checked, the six-month build gate on licence and cost data, and an explicit list of what this index gets wrong. - [https://lattice.kunjshah.in/]#start-here — Two questions that narrow 9 layers to the 3 worth reading first. - [https://lattice.kunjshah.in/fix] — Start from a symptom (slow, expensive, wrong answers, unreliable agent): an ordered checklist through the stack. - [https://lattice.kunjshah.in/compare] — Head-to-head comparisons with a recommendation and the conditions it depends on. - [https://lattice.kunjshah.in/roles] — The index cut by engineering specialisation rather than stack layer: what each role is accountable for. ## Common tasks - Why is my LLM app slow? → https://lattice.kunjshah.in/fix/llm-app-too-slow - Why is my LLM bill so high? → https://lattice.kunjshah.in/fix/llm-costs-too-high - Why does my LLM give wrong answers? → https://lattice.kunjshah.in/fix/llm-wrong-answers - Why does my AI agent keep failing? → https://lattice.kunjshah.in/fix/ai-agent-unreliable - How do I know if my LLM changes actually helped? → https://lattice.kunjshah.in/fix/measure-llm-changes - Why does my RAG system hallucinate? → https://lattice.kunjshah.in/fix/rag-hallucinations - How do I stop PII showing up in LLM traces and prompts? → https://lattice.kunjshah.in/fix/pii-in-llm-traces - Choosing an inference engine → https://lattice.kunjshah.in/compare/inference-runtimes - What a ml platform role owns → https://lattice.kunjshah.in/roles/platform - What a ai infrastructure role owns → https://lattice.kunjshah.in/roles/serving - What a data & retrieval role owns → https://lattice.kunjshah.in/roles/data - What a applied engineering role owns → https://lattice.kunjshah.in/roles/applied - What a production & governance role owns → https://lattice.kunjshah.in/roles/production - Tracing and eval in one tool → https://lattice.kunjshah.in/compare/llm-observability - One gateway across many providers → https://lattice.kunjshah.in/routing-gateways ## Policies - Contact → https://lattice.kunjshah.in/contact - About → https://lattice.kunjshah.in/about - Privacy → https://lattice.kunjshah.in/privacy - Returns → https://lattice.kunjshah.in/returns ## Essays - [Choosing an inference runtime](https://lattice.kunjshah.in/blog/choosing-an-inference-runtime) — The wrong runtime costs you throughput you never knew you were leaving on the table. The right one is usually decided by traffic shape, not model size. - [The gateway is the product](https://lattice.kunjshah.in/blog/the-gateway-is-the-product) — The interesting engineering is not calling a model. It is deciding which model, failing over safely, and staying inside a budget while you do it. - [Fix the ranking, not the prompt](https://lattice.kunjshah.in/blog/fix-the-ranking-not-the-prompt) — When a RAG system gives a bad answer, the instinct is to rewrite the prompt. Nine times out of ten the prompt is fine and the context was never there. - [Evals are the asset, not the logging](https://lattice.kunjshah.in/blog/evals-are-the-asset) — Traces tell you what happened. Evals tell you whether it got better. Most teams build the first and mistake it for the second. - [Agents need a durable host](https://lattice.kunjshah.in/blog/agents-need-a-durable-host) — The loop is a weekend project. The part where it keeps working at three in the morning is the actual product, and almost nobody budgets for it. - [Prompt, or fine-tune?](https://lattice.kunjshah.in/blog/prompt-or-finetune) — Almost every 'the model does not know this' problem is a retrieval problem wearing a hat. Here is how to tell the difference before you spend a week on a training run. - [Where guardrails belong](https://lattice.kunjshah.in/blog/where-guardrails-belong) — Safety is not a layer you add. It is a set of controls at specific points in a request's path, and each one catches a different class of failure. - [Context is a budget, not a container](https://lattice.kunjshah.in/blog/context-is-a-budget) — The instinct is to keep everything in case it is needed. The models are not filing systems, and past a certain length the middle of your context is doing nothing but costing money. - [Build a cost model before you optimise](https://lattice.kunjshah.in/blog/the-cost-model) — Teams optimise the thing they can see in the dashboard. The dashboard shows tokens. The number that matters is cost per successful outcome, and nobody is measuring it. - [Choosing a model is a routing problem](https://lattice.kunjshah.in/blog/choosing-a-model) — There is no best model. There is a best model for a task, at a quality bar, within a latency and cost budget — and that decision is a routing policy, not a benchmark. - [Observability is not logging](https://lattice.kunjshah.in/blog/observability-is-not-logging) — A log line with a status code is not a trace. The difference is whether it can reconstruct a decision after the model that made it has been replaced twice. ## Comparisons - [vLLM vs SGLang vs TGI vs llama.cpp](https://lattice.kunjshah.in/compare/inference-runtimes) — Start with vLLM. Benchmark SGLang against it if more than a few hundred requests share a long system prompt, because prefix reuse compounds and the gap is not marginal. Use llama.cpp when the constraint is hardware rather than throughput. Reach for TGI only when you specifically need multi-GPU tensor parallelism inside an existing Hugging Face setup. - [Langfuse vs LangSmith vs Braintrust vs Phoenix](https://lattice.kunjshah.in/compare/llm-observability) — If you cannot send traces outside your infrastructure, self-host Langfuse and stop deliberating. Otherwise pick on evaluation rather than tracing, because tracing is table stakes and evaluation is what you will actually rely on. Whatever you choose, make sure traces carry the full request and response — that is the input to every future decision. - [pgvector vs Qdrant vs Pinecone vs Chroma](https://lattice.kunjshah.in/compare/vector-databases) — Start on pgvector unless you have a specific reason not to. It removes a whole class of consistency problem, and if your retrieval is joining against rows you already have — users, permissions, tenancy — the relational model is simply the right one. Move to Qdrant or Pinecone when vector search stops being a side feature and starts being the workload. - [Temporal vs Inngest vs Trigger.dev vs Restate](https://lattice.kunjshah.in/compare/durable-workflows) — If your stack is TypeScript, Inngest or Trigger.dev will be live in a day and cover 90% of agent workloads. If you are polyglot or durability is the centre of your business, Temporal is the mature answer and you should accept the adoption cost. Restate is the one to watch if low-latency stateful endpoints matter more than ecosystem size. - [LiteLLM vs Portkey vs Cloudflare AI Gateway](https://lattice.kunjshah.in/compare/llm-gateways) — Self-host LiteLLM if you need breadth and control; it is the least committal. Choose Portkey if you want a managed product with guardrails included and are comfortable with a SaaS dependency. Cloudflare AI Gateway is the pick when latency and edge caching dominate, and Envoy AI Gateway when you need a gateway that never sees traffic leave your network. - [Instructor vs JSON mode vs constrained decoding](https://lattice.kunjshah.in/compare/structured-output) — Use a library like Instructor for extraction work: validation, typed retry and partial streaming are worth more than an absolute guarantee. Reach for true constrained decoding — Guidance or Outlines — when downstream code cannot tolerate a malformed response at all, because there the guarantee is worth the reduced flexibility. Treat a provider's JSON mode as a convenience rather than a contract, and validate regardless. - [Gateway vs guardrails vs evals: what to build first](https://lattice.kunjshah.in/compare/gateway-guardrails-evals) — For most teams, evals first. A gateway makes calls cheaper and more reliable, and guardrails make individual responses safer, but neither tells you whether the system is getting better — and every later change to routing, prompts or retrieval needs that answer to be judged at all. Put the gateway in early if you already run more than one provider, because retrofitting a proxy into every call site is tedious. Add guardrails for the specific failures your evals surface, not the ones you imagine. - [Retrieval vs fine-tuning vs prompt optimisation: fixing wrong answers](https://lattice.kunjshah.in/compare/retrieval-finetuning-prompting) — Diagnose before choosing. If the right fact was never in the context, that is a retrieval problem and no amount of prompting or training fixes it. If the fact was there and the model ignored or misused it, optimise the prompt first, because it is cheap to try and cheap to undo. Fine-tune last, for narrow and stable tasks where prompting has measurably plateaued — it is the only one of the three that leaves you with a new artefact to serve and retrain. - [Self-hosting vs routing vs measuring: cutting inference cost](https://lattice.kunjshah.in/compare/cutting-inference-cost) — Measure first. Per-request cost attribution is the cheapest of the three and the only one with no quality risk, and it often shows a few features or prompts dominating spend — which a shorter prompt can fix without new infrastructure. Route next, but only once an eval can tell you what the cheaper model loses. Self-host last, when measured and sustained utilisation makes GPU-hours cheaper than tokens; below that line it raises the bill. - [LangChain vs LlamaIndex vs Pydantic AI vs OpenAI Agents SDK](https://lattice.kunjshah.in/compare/agent-frameworks) — Choose on the shape of the job. LlamaIndex when retrieval and connectors are the product; Pydantic AI when schema-safe tool calls are the risk; OpenAI Agents SDK when you want a small loop with handoffs and guardrails built in; LangChain when you need the widest integration surface and will pay the abstraction tax. Whichever you pick, run the loop on a durable host and trace every tool call — no framework replaces those. - [NeMo Guardrails vs Guardrails AI vs Presidio vs Invariant](https://lattice.kunjshah.in/compare/guardrails-platforms) — Start with the failure you have seen. Presidio when PII in prompts or logs is the incident; Guardrails AI when bad JSON or schema violations are the incident; NeMo when the model keeps drifting off-topic across turns; Invariant when you need policy enforced on every hop with code reviewable rules. Whatever you pick, redact before traces land in a third-party observability tool — guardrails on the model do not fix logging. ## Glossary - [Paged attention](https://lattice.kunjshah.in/glossary/paged-attention) — Managing the KV cache in fixed-size blocks rather than one contiguous allocation, so memory is not fragmented and sequences can share a batch. - [KV cache](https://lattice.kunjshah.in/glossary/kv-cache) — The saved attention keys and values from tokens already processed, so the model does not recompute them for every new token. - [Continuous batching](https://lattice.kunjshah.in/glossary/continuous-batching) — Adding new sequences to a batch as others finish, rather than waiting for the whole batch to complete. - [Prefix caching](https://lattice.kunjshah.in/glossary/prefix-caching) — Reusing the KV cache for a prompt prefix that has already been processed, instead of recomputing it. - [Quantisation](https://lattice.kunjshah.in/glossary/quantisation) — Storing and computing model weights in fewer bits than they were trained in, to trade accuracy for memory and speed. - [Tensor parallelism](https://lattice.kunjshah.in/glossary/tensor-parallelism) — Splitting a single layer's computation across several GPUs, so every token passes through all of them. - [Speculative decoding](https://lattice.kunjshah.in/glossary/speculative-decoding) — Using a small draft model to propose several tokens at once, which a large model then verifies in a single pass. - [Time to first token](https://lattice.kunjshah.in/glossary/time-to-first-token) — The delay between sending a request and receiving the first token of the response. - [Throughput and latency](https://lattice.kunjshah.in/glossary/throughput) — Throughput is tokens per second across the whole deployment; latency is how long one request takes. They trade directly against each other. - [Constrained decoding](https://lattice.kunjshah.in/glossary/constrained-output) — Masking the token space during generation so the output can only be a valid instance of a schema, grammar or regex. - [Validation and repair](https://lattice.kunjshah.in/glossary/schema-enforcement) — Generating freely, then validating the output against a schema and retrying with the error fed back. - [Model router](https://lattice.kunjshah.in/glossary/model-router) — A layer that chooses which model answers a given request, from signals known before the model is called. - [The escalation trap](https://lattice.kunjshah.in/glossary/escalation-trap) — Routing cheap-first and retrying on a bigger model when the answer looks bad, which usually costs more than routing correctly up front. - [Semantic cache](https://lattice.kunjshah.in/glossary/semantic-cache) — Reusing a previous response when a new request is sufficiently similar in meaning, skipping the model call entirely. - [Circuit breaker](https://lattice.kunjshah.in/glossary/circuit-breaker) — A per-provider, per-model gate that stops sending traffic somewhere after repeated failures, then probes for recovery. - [Provider prompt caching](https://lattice.kunjshah.in/glossary/prompt-caching) — A discount providers apply when a request repeats a prefix they have already processed. - [Embedding](https://lattice.kunjshah.in/glossary/embedding) — A vector representation of text such that semantically similar inputs land near each other. - [Chunking](https://lattice.kunjshah.in/glossary/chunking) — Splitting documents into retrievable pieces, which is where RAG quality is mostly won and lost. - [Hybrid search](https://lattice.kunjshah.in/glossary/hybrid-search) — Combining lexical retrieval (BM25) with vector retrieval, so exact matches and semantic matches are both found. - [Reranking](https://lattice.kunjshah.in/glossary/reranking) — Re-scoring retrieved candidates with a slower, more accurate model before the generator sees them. - [Reciprocal rank fusion](https://lattice.kunjshah.in/glossary/reciprocal-rank-fusion) — Merging ranked lists from several retrievers by summing the reciprocal of each result's rank, rather than comparing scores. - [Vector index](https://lattice.kunjshah.in/glossary/vector-index) — An approximate structure — usually HNSW or an IVF variant — that finds nearest neighbours without scanning every vector. - [Recall@k](https://lattice.kunjshah.in/glossary/recall-at-k) — Of the documents that should answer a query, the fraction that appear in the top k retrieved. - [Context window](https://lattice.kunjshah.in/glossary/context-window) — The number of tokens a model can consider at once — a ceiling on what you can send, not a target for what you should. - [LoRA](https://lattice.kunjshah.in/glossary/lora) — Training small low-rank matrices alongside frozen weights, so you can adapt a large model without touching it. - [Supervised fine-tuning](https://lattice.kunjshah.in/glossary/sft) — Training on curated input/output pairs so the model adopts a specific task or response style. - [Direct preference optimisation](https://lattice.kunjshah.in/glossary/dpo) — Aligning a model to human preferences without an explicit reward model, by optimising on chosen-versus-rejected pairs. - [RLHF](https://lattice.kunjshah.in/glossary/rlhf) — Reinforcement learning from human feedback: training a reward model from preferences, then optimising the policy against it. - [GRPO](https://lattice.kunjshah.in/glossary/grpo) — Group relative policy optimisation — a reinforcement-learning method that compares a group of sampled answers rather than learning a separate reward model. - [Distillation](https://lattice.kunjshah.in/glossary/distillation) — Training a smaller model against the outputs of a larger one, to get most of the behaviour at a fraction of the cost. - [ZeRO](https://lattice.kunjshah.in/glossary/zero) — Sharding optimiser states, gradients and parameters across data-parallel workers to fit a larger model on the same GPUs. - [Tool calling](https://lattice.kunjshah.in/glossary/tool-calling) — A model emitting a structured request to call a function, which your code executes and returns the result of. - [Agent loop](https://lattice.kunjshah.in/glossary/agent-loop) — The control structure where a model decides an action, it is executed, and the result feeds the next turn until the task is done. - [Model Context Protocol](https://lattice.kunjshah.in/glossary/mcp) — An open protocol for exposing tools, resources and prompts to models, so integrations are written once rather than per model provider. - [Budget ceiling](https://lattice.kunjshah.in/glossary/budget-ceiling) — A hard limit on turns, tokens or wall-clock time, enforced by the host rather than requested in the prompt. - [Durable execution](https://lattice.kunjshah.in/glossary/durable-execution) — A workflow engine that checkpoints progress so work resumes after a crash, rather than restarting from nothing. - [Idempotency key](https://lattice.kunjshah.in/glossary/idempotency-key) — A client-supplied identifier that lets a server recognise a repeated request and return the original result rather than acting twice. - [Prompt injection](https://lattice.kunjshah.in/glossary/prompt-injection) — An attacker getting instructions into your context that your model then follows as if they came from you. - [Guardrail](https://lattice.kunjshah.in/glossary/guardrail) — Any control that keeps a model from doing something you have decided it must not do. - [Safety classifier](https://lattice.kunjshah.in/glossary/llama-guard) — A model, distinct from the one you are serving, that labels content as safe or unsafe. - [PII redaction](https://lattice.kunjshah.in/glossary/pii-redaction) — Detecting and masking personal data before it enters a system you do not control. - [Few-shot prompting](https://lattice.kunjshah.in/glossary/few-shot) — Including worked examples in the prompt instead of only describing what you want. - [System prompt](https://lattice.kunjshah.in/glossary/system-prompt) — The instruction block that frames a conversation, separate from the user's messages. - [Golden set](https://lattice.kunjshah.in/glossary/golden-set) — A small, curated set of real cases, with expected behaviour written down, used to tell whether a change helped. - [LLM-as-judge](https://lattice.kunjshah.in/glossary/llm-as-judge) — Using a model to score output against a written rubric, in place of a human. - [Regression gate](https://lattice.kunjshah.in/glossary/eval-gate) — A check in CI that blocks a merge when a change makes a scored dimension worse. - [Shadow traffic](https://lattice.kunjshah.in/glossary/shadow-traffic) — Replaying a sample of production requests against a candidate model without serving the answers. - [Cost per successful outcome](https://lattice.kunjshah.in/glossary/cost-per-outcome) — Total spend divided by the count of requests that passed whatever quality bar you can define. - [Open weight vs open source](https://lattice.kunjshah.in/glossary/open-weight) — Open weights means you can download and run the model. Open source additionally means the training data, training code and licence permit modification and redistribution. - [Mixture of experts](https://lattice.kunjshah.in/glossary/mixture-of-experts) — A model with many parameter blocks, of which only a small number are activated per token. - [Hallucination](https://lattice.kunjshah.in/glossary/hallucination) — Fluent, confident output that is not supported by the input or by evidence. ## [Inference & Serving](https://lattice.kunjshah.in/inference-serving) Turns weights into throughput. - [vLLM](https://lattice.kunjshah.in/inference-serving/vllm) — Paged-attention inference engine with an OpenAI-compatible server. - Use when: General GPU serving under bursty, high-concurrency traffic. - Skip when: Traffic is steady and low-concurrency, so batches never fill. - Substitutes: SGLang (Inference), TensorRT-LLM (Inference), Text Generation Inference (Inference), Triton Inference Server (Inference), PowerInfer (Inference) - [SGLang](https://lattice.kunjshah.in/inference-serving/sglang) — Structured generation runtime built on RadixAttention prefix reuse. - Use when: A large share of requests share a long prefix, or you need constrained decoding. - Skip when: You want the fewest moving parts and vLLM already clears your bar. - Substitutes: vLLM (Inference), TensorRT-LLM (Inference), Text Generation Inference (Inference) - [llama.cpp](https://lattice.kunjshah.in/inference-serving/llama-cpp) — Portable CPU/GPU inference for quantized GGUF models. - Use when: The constraint is hardware, not throughput — no GPU, or an edge box. - Skip when: You need maximum concurrent throughput on datacentre GPUs. - Substitutes: Ollama (Inference), LM Studio (Inference), llamafile (Inference), KoboldCpp (Inference), PowerInfer (Inference) - [Ollama](https://lattice.kunjshah.in/inference-serving/ollama) — Local model runner with a single-binary distribution and HTTP API. - Use when: Local model access for a developer or small team, with no setup. - Skip when: You need production throughput or a stable server API. - Substitutes: llama.cpp (Inference), LM Studio (Inference), llamafile (Inference), KoboldCpp (Inference) - [TensorRT-LLM](https://lattice.kunjshah.in/inference-serving/tensorrt-llm) — NVIDIA-optimized inference with inflight batching and FP8. - Use when: You are committed to NVIDIA hardware and need the last of the throughput. - Skip when: Hardware portability matters, or you have no GPUs to tune against. - Substitutes: vLLM (Inference), SGLang (Inference) - [Text Generation Inference](https://lattice.kunjshah.in/inference-serving/text-generation-inference) — Hugging Face production server for LLMs with tensor parallelism. - Use when: Multi-GPU tensor parallelism inside an existing Hugging Face stack. - Skip when: The model fits on one GPU, where this is overkill. - Substitutes: vLLM (Inference), SGLang (Inference), Triton Inference Server (Inference) - [LM Studio](https://lattice.kunjshah.in/inference-serving/lm-studio) — Desktop app for running and serving local models with an OpenAI API. - Use when: A desktop GUI for running and serving local models, aimed at one operator. - Skip when: You need headless, multi-tenant deployment. - Substitutes: Ollama (Inference), llama.cpp (Inference) - [Triton Inference Server](https://lattice.kunjshah.in/inference-serving/triton-inference-server) — Multi-framework inference server for ONNX, TensorRT and Python. - Use when: One endpoint serving ONNX, TensorRT and Python models side by side. - Skip when: You serve one model family and want peak LLM throughput. - Substitutes: vLLM (Inference), Text Generation Inference (Inference) - [llamafile](https://lattice.kunjshah.in/inference-serving/llamafile) — Packages a model and its runtime into a single executable. - Use when: Shipping a model as a single executable with no install step. - Skip when: You need batching, multi-GPU serving or a production HTTP server. - Substitutes: llama.cpp (Inference), Ollama (Inference) - [KoboldCpp](https://lattice.kunjshah.in/inference-serving/koboldcpp) — GGUF inference with a focused text-adventure UI and API. - Use when: Local generation with a UI tuned for interactive, long-form use. - Skip when: You need headless serving at scale. - Substitutes: llama.cpp (Inference), Ollama (Inference) - [PowerInfer](https://lattice.kunjshah.in/inference-serving/powerinfer) — CPU-first runtime that offloads hot paths to the GPU. - Use when: CPU-first serving where the GPU accelerates rather than carries the load. - Skip when: You have GPUs to spare — it trades peak throughput for reach. - Substitutes: llama.cpp (Inference), vLLM (Inference) - [Marlin](https://lattice.kunjshah.in/inference-serving/marlin) — Quantised GEMM kernels for near-GPU speed at 4-bit. ## [Routing & Gateways](https://lattice.kunjshah.in/routing-gateways) Decides which model answers, and what it costs. - [LiteLLM](https://lattice.kunjshah.in/routing-gateways/litellm) — OpenAI-format proxy translating across 100+ model providers. - Use when: The widest provider coverage, deployed the least committal way. - Skip when: You want routing policy to live in a SaaS console. - Substitutes: Portkey (Routing), OpenRouter (Routing), Envoy AI Gateway (Routing), Bifrost (Routing), Cloudflare AI Gateway (Routing) - [Portkey](https://lattice.kunjshah.in/routing-gateways/portkey) — AI gateway with routing, caching and guardrails in one layer. - Use when: A managed gateway with guardrails and analytics already bundled. - Skip when: Prompts cannot leave your infrastructure. - Substitutes: LiteLLM (Routing), Cloudflare AI Gateway (Routing), TrueFoundry (Routing) - [Cloudflare AI Gateway](https://lattice.kunjshah.in/routing-gateways/cloudflare-ai-gateway) — Edge gateway adding caching, retries and rate limiting. - Use when: Edge caching and latency dominate, on Cloudflare already. - Skip when: Requests must not transit a third party's network. - Substitutes: LiteLLM (Routing), Portkey (Routing), Envoy AI Gateway (Routing) - [OpenRouter](https://lattice.kunjshah.in/routing-gateways/openrouter) — Single API key and billing across many hosted models. - Use when: One key and one bill across many hosted models, while experimenting. - Skip when: You need self-hosting or per-provider SLAs. - Substitutes: Martian (Routing), LiteLLM (Routing) - [Martian](https://lattice.kunjshah.in/routing-gateways/martian) — Model router that optimizes for quality, cost and latency. - Use when: Routing tuned on your own traffic rather than a public benchmark. - Skip when: You want the routing rule to be plain config you can read. - Substitutes: OpenRouter (Routing), RouteLLM (Routing), Not Diamond (Routing) - [Envoy AI Gateway](https://lattice.kunjshah.in/routing-gateways/envoy-ai-gateway) — CNCF-track gateway for LLM traffic on an Envoy data plane. - Use when: A Kubernetes shop that already runs Envoy and needs the gateway in-cluster. - Skip when: You want a gateway configurable without a data plane. - Substitutes: LiteLLM (Routing), Cloudflare AI Gateway (Routing) - [Bifrost](https://lattice.kunjshah.in/routing-gateways/bifrost) — High-throughput LLM gateway with drop-in OpenAI compatibility. - Use when: A high-throughput OpenAI-compatible proxy with a small footprint. - Skip when: You need the breadth of a full routing stack. - Substitutes: LiteLLM (Routing) - [RouteLLM](https://lattice.kunjshah.in/routing-gateways/routellm) — Learned router that cuts cost by matching difficulty to model size. - Use when: Cutting cost by matching query difficulty to model size, with a learned router. - Skip when: You cannot evaluate the quality you would be trading away. - Substitutes: Martian (Routing), Not Diamond (Routing) - [Not Diamond](https://lattice.kunjshah.in/routing-gateways/not-diamond) — Routing and prompt optimisation tuned on your own traffic. - Use when: Routing and prompt optimisation tuned on your own traffic. - Skip when: Self-hosting or auditability is a requirement. - Substitutes: Martian (Routing), RouteLLM (Routing) - [TrueFoundry](https://lattice.kunjshah.in/routing-gateways/truefoundry) — Gateway plus observability and guardrails as one deployment. - Use when: Gateway, observability and guardrails as one managed deployment. - Skip when: You want to adopt the pieces independently. - Substitutes: Portkey (Routing), Braintrust (Evals) ## [Retrieval & Vector Stores](https://lattice.kunjshah.in/retrieval-vector-stores) Holds the embeddings, returns the few that matter. - [pgvector](https://lattice.kunjshah.in/retrieval-vector-stores/pgvector) — Vector similarity search as a Postgres extension. - Use when: Under a few million vectors, or when retrieval joins rows you already have. - Skip when: Vector search has become the workload rather than a side feature. - Substitutes: Qdrant (Retrieval), Chroma (Retrieval), Milvus (Retrieval), Weaviate (Retrieval) - [Qdrant](https://lattice.kunjshah.in/retrieval-vector-stores/qdrant) — Rust vector database with rich filtering payloads. - Use when: Vector-first workloads with heavy metadata filtering at scale. - Skip when: You want no new datastore to operate. - Substitutes: pgvector (Retrieval), Weaviate (Retrieval), Chroma (Retrieval), Pinecone (Retrieval), Milvus (Retrieval), Turbopuffer (Retrieval), Elasticsearch (Retrieval) - [Weaviate](https://lattice.kunjshah.in/retrieval-vector-stores/weaviate) — Graph-aware vector database with hybrid and multi-vector search. - Use when: Hybrid or multi-vector search with a graph-shaped data model. - Skip when: Your data is genuinely relational and joins matter more than vectors. - Substitutes: pgvector (Retrieval), Qdrant (Retrieval), Milvus (Retrieval), Elasticsearch (Retrieval), Vespa (Retrieval) - [Chroma](https://lattice.kunjshah.in/retrieval-vector-stores/chroma) — Embeddings database designed for fast prototyping. - Use when: Getting a prototype working in an afternoon. - Skip when: You need durability, scaling or concurrent writes. - Substitutes: Qdrant (Retrieval), pgvector (Retrieval) - [Pinecone](https://lattice.kunjshah.in/retrieval-vector-stores/pinecone) — Fully managed vector database with serverless scaling. - Use when: Zero operational work, and vector search is not your core competence. - Skip when: Residency, cost predictability or a query language you control. - Substitutes: Turbopuffer (Retrieval), Milvus (Retrieval), Qdrant (Retrieval) - [Milvus](https://lattice.kunjshah.in/retrieval-vector-stores/milvus) — Cloud-native vector database supporting billion-scale indexes. - Use when: Billion-scale indexes with a cloud-native deployment. - Skip when: You need a small, comprehensible system. - Substitutes: pgvector (Retrieval), Qdrant (Retrieval), Pinecone (Retrieval), Weaviate (Retrieval) - [Turbopuffer](https://lattice.kunjshah.in/retrieval-vector-stores/turbopuffer) — Vector search engine tuned for high-recall retrieval workloads. - Use when: High-recall retrieval at volume, without operating the index yourself. - Skip when: You need to tune the index directly. - Substitutes: Pinecone (Retrieval), Qdrant (Retrieval) - [Unstructured](https://lattice.kunjshah.in/retrieval-vector-stores/unstructured) — Preprocessing library that partitions raw documents for indexing. - Use when: Partitioning messy documents into retrievable chunks before indexing. - Skip when: Your input is already structured. - Substitutes: Rerankers (Retrieval), LangChain Text Splitters (Retrieval), Docling (Retrieval), LlamaParse (Retrieval) - [Elasticsearch](https://lattice.kunjshah.in/retrieval-vector-stores/elasticsearch) — BM25 plus dense vectors in one index, with hybrid ranking. - Use when: One index carrying BM25 and dense vectors, with ranking you can tune. - Skip when: You need a permissive licence — the Elastic licence is not OSI-approved. - Substitutes: Weaviate (Retrieval), Vespa (Retrieval), Qdrant (Retrieval) - [Vespa](https://lattice.kunjshah.in/retrieval-vector-stores/vespa) — Search engine built for ranking-heavy retrieval at scale. - Use when: Ranking-heavy retrieval where the ranking function is the product. - Skip when: You want a small, low-operations component. - Substitutes: Weaviate (Retrieval), Elasticsearch (Retrieval) - [Rerankers](https://lattice.kunjshah.in/retrieval-vector-stores/rerankers) — Purpose-built cross-encoders for the second retrieval stage. - Use when: A cross-encoder in the second retrieval stage, to lift precision cheaply. - Skip when: You have no way to measure whether recall actually improved. - Substitutes: Jina AI (Retrieval), Unstructured (Retrieval) - [Jina AI](https://lattice.kunjshah.in/retrieval-vector-stores/jina-ai) — Multimodal embeddings and a reranker behind one endpoint. - Use when: Multimodal embeddings and a reranker behind one endpoint. - Skip when: Embeddings must stay in-house. - Substitutes: Rerankers (Retrieval) - [LangChain Text Splitters](https://lattice.kunjshah.in/retrieval-vector-stores/langchain-text-splitters) — Document loaders and splitters for the ingest stage. - Use when: Document loaders and splitters, if you are already in that ecosystem. - Skip when: Structure matters — fixed-size splitting is rarely the right boundary. - Substitutes: Unstructured (Retrieval), Docling (Retrieval) - [Docling](https://lattice.kunjshah.in/retrieval-vector-stores/docling) — Layout-aware parsing that preserves table and heading structure. - Use when: Layout-aware parsing that preserves tables and heading structure. - Skip when: Your documents are plain text. - Substitutes: Unstructured (Retrieval), LangChain Text Splitters (Retrieval), LlamaParse (Retrieval) - [LlamaParse](https://lattice.kunjshah.in/retrieval-vector-stores/llamaparse) — Managed parsing for PDFs, especially the ugly ones. - Use when: Managed parsing for PDFs, especially the ugly ones. - Skip when: Document contents cannot leave your infrastructure. - Substitutes: Unstructured (Retrieval), Docling (Retrieval) ## [Fine-tuning & Training](https://lattice.kunjshah.in/fine-tuning) Produces the weights that layer 1 serves. - [Unsloth](https://lattice.kunjshah.in/fine-tuning/unsloth) — Hand-written kernels that cut LoRA memory and time sharply. - Use when: LoRA fine-tuning on a single GPU, with memory and time roughly halved. - Skip when: You need a training stack you can audit line by line. - Substitutes: Axolotl (Training), PEFT (Training) - [Axolotl](https://lattice.kunjshah.in/fine-tuning/axolotl) — Configuration-driven fine-tuning across common architectures. - Use when: Configuration-driven fine-tuning across many architectures. - Skip when: You need to hand-tune the training loop itself. - Substitutes: Unsloth (Training), LLaMA-Factory (Training) - [LLaMA-Factory](https://lattice.kunjshah.in/fine-tuning/llama-factory) — Unified interface for SFT, DPO and RLHF on open models. - Use when: One interface for SFT, DPO and RLHF across open models. - Skip when: You want a minimal, readable training script. - Substitutes: Axolotl (Training), TRL (Training), torchtune (Training) - [PEFT](https://lattice.kunjshah.in/fine-tuning/peft) — Parameter-efficient fine-tuning methods such as LoRA and QLoRA. - Use when: Parameter-efficient fine-tuning — LoRA and variants — on limited hardware. - Skip when: You need to change the model's capabilities, not bolt on an adapter. - Substitutes: Unsloth (Training), TRL (Training) - [TRL](https://lattice.kunjshah.in/fine-tuning/trl) — Hugging Face library of post-training trainers for SFT, DPO and GRPO. - Use when: Post-training trainers for SFT, DPO and GRPO on top of any PEFT setup. - Skip when: You are training from scratch rather than aligning an existing model. - Substitutes: LLaMA-Factory (Training), PEFT (Training), torchtune (Training), Hugging Face TRL (Training) - [DeepSpeed](https://lattice.kunjshah.in/fine-tuning/deepspeed) — ZeRO sharding and pipeline parallelism for large-scale training. - Use when: ZeRO sharding and pipeline parallelism across many GPUs. - Skip when: The model fits on one device — the complexity buys nothing. - Substitutes: Megatron-LM (Training) - [Megatron-LM](https://lattice.kunjshah.in/fine-tuning/megatron-lm) — Tensor and pipeline parallel training for multi-GPU clusters. - Use when: Tensor and pipeline parallel pretraining at cluster scale. - Skip when: You are fine-tuning rather than pretraining. - Substitutes: DeepSpeed (Training) - [torchtune](https://lattice.kunjshah.in/fine-tuning/torchtune) — PyTorch-native recipes for fine-tuning and aligning open models. - Use when: PyTorch-native recipes that you can read and modify. - Skip when: You want the breadth of a config-driven stack. - Substitutes: TRL (Training), LLaMA-Factory (Training) - [Hugging Face TRL](https://lattice.kunjshah.in/fine-tuning/hugging-face-trl) — Preference and reward optimisation on top of any PEFT setup. - Use when: Preference and reward optimisation hosted, without a cluster. - Skip when: Training data cannot be uploaded to a third party. - Substitutes: TRL (Training) - [Colab](https://lattice.kunjshah.in/fine-tuning/colab) — Hosted GPUs for small runs and quick experiments. - Use when: A disposable GPU for a quick experiment, with no setup. - Skip when: Anything reproducible — a notebook is not a training pipeline. - Substitutes: Replicate (Training) - [Replicate](https://lattice.kunjshah.in/fine-tuning/replicate) — Hosted fine-tuning and deployment for open models. - Use when: Hosted fine-tuning and deployment of an open model, billed per run. - Skip when: You need custom training code or strict residency. - Substitutes: Colab (Training), Weights & Biases Launch (Training), Modal (Workflows) - [Weights & Biases Launch](https://lattice.kunjshah.in/fine-tuning/weights-biases-launch) — Managed training runs with sweeps and artifact tracking. - Use when: Managed training runs with sweeps and artifact tracking wired up. - Skip when: You want training infrastructure you run yourself. - Substitutes: Replicate (Training), Weights & Biases (Evals) ## [Agent Frameworks](https://lattice.kunjshah.in/agent-frameworks) Turns a model call into a multi-step program. - [LangChain](https://lattice.kunjshah.in/agent-frameworks/langchain) — Composable abstractions for model calls, tools and state. - Use when: Tool use and state, in an ecosystem where the answers already exist. - Skip when: You want a thin, legible core — this is a large surface. - Substitutes: LlamaIndex (Agents), Pydantic AI (Agents), AutoGen (Agents), CrewAI (Agents), Semantic Kernel (Agents) - [LlamaIndex](https://lattice.kunjshah.in/agent-frameworks/llamaindex) — Data-centric framework for retrieval and agent workflows. - Use when: Data-centric work where retrieval is the centre of the problem. - Skip when: You need agent orchestration more than data access. - Substitutes: LangChain (Agents), Pydantic AI (Agents) - [Pydantic AI](https://lattice.kunjshah.in/agent-frameworks/pydantic-ai) — Type-safe agent framework that leans on Pydantic models. - Use when: Type-safe agents where schema correctness is the priority. - Skip when: You want a batteries-included ecosystem. - Substitutes: LangChain (Agents), LlamaIndex (Agents), Semantic Kernel (Agents), Agno (Agents), smolagents (Agents) - [AutoGen](https://lattice.kunjshah.in/agent-frameworks/autogen) — Microsoft research project for conversable multi-agent systems. - Use when: Research into conversable multi-agent systems. - Skip when: You need something stable and documented for production. - Substitutes: CrewAI (Agents), LangChain (Agents) - [CrewAI](https://lattice.kunjshah.in/agent-frameworks/crewai) — Role-based orchestration where agents collaborate as a crew. - Use when: Role-based orchestration where agents are framed as a team. - Skip when: You want direct control over the loop itself. - Substitutes: LangChain (Agents), AutoGen (Agents) - [Semantic Kernel](https://lattice.kunjshah.in/agent-frameworks/semantic-kernel) — Microsoft SDK for embedding AI steps into .NET and Python apps. - Use when: Embedding AI steps into an existing .NET estate. - Skip when: Your stack is not .NET. - Substitutes: LangChain (Agents), Pydantic AI (Agents) - [Mastra](https://lattice.kunjshah.in/agent-frameworks/mastra) — TypeScript agent framework with typed workflows and evals. - Use when: TypeScript agents with typed workflows and evals built in. - Skip when: You are working in Python. - Substitutes: Vercel AI SDK (Agents), OpenAI Agents SDK (Agents) - [OpenAI Agents SDK](https://lattice.kunjshah.in/agent-frameworks/openai-agents-sdk) — Lightweight primitives for handoffs, guardrails and tracing. - Use when: A small, legible set of primitives for handoffs and guardrails. - Skip when: You need multi-provider abstraction at every layer. - Substitutes: Mastra (Agents), Claude Agent SDK (Agents), Letta (Agents), Vercel AI SDK (Agents), smolagents (Agents) - [Agno](https://lattice.kunjshah.in/agent-frameworks/agno) — Minimal agent runtime centered on model-agnostic tool interfaces. - Use when: A minimal agent runtime with model-agnostic tool interfaces. - Skip when: You need the wider ecosystem around it. - Substitutes: smolagents (Agents), Pydantic AI (Agents) - [Claude Agent SDK](https://lattice.kunjshah.in/agent-frameworks/claude-agent-sdk) — Anthropic's toolkit for building agents with tool use and hooks. - Use when: Building agents against Anthropic models, with tool use and hooks. - Skip when: You need provider independence at the framework layer. - Substitutes: OpenAI Agents SDK (Agents), Letta (Agents) - [Letta](https://lattice.kunjshah.in/agent-frameworks/letta) — Agent runtime with persistent, editable memory as a first-class primitive. - Use when: Agents whose memory is a first-class, inspectable and editable state. - Skip when: You want memory to be a detail the framework handles for you. - Substitutes: Claude Agent SDK (Agents), smolagents (Agents), OpenAI Agents SDK (Agents) - [smolagents](https://lattice.kunjshah.in/agent-frameworks/smolagents) — Minimal code-first agent loop from the Hugging Face team. - Use when: Reading the whole agent loop in a page — it is deliberately tiny. - Skip when: You need a rich tool ecosystem around it. - Substitutes: Pydantic AI (Agents), OpenAI Agents SDK (Agents), Agno (Agents), Letta (Agents) - [Vercel AI SDK](https://lattice.kunjshah.in/agent-frameworks/vercel-ai-sdk) — Provider-agnostic building blocks for streaming apps and agents. - Use when: Streaming model calls in a TypeScript app, provider-agnostically. - Skip when: You need agent orchestration rather than model calls. - Substitutes: Mastra (Agents), OpenAI Agents SDK (Agents) ## [Workflow Orchestration](https://lattice.kunjshah.in/workflow-orchestration) Survives the retries, the waits and the crashes. - [Temporal](https://lattice.kunjshah.in/workflow-orchestration/temporal) — Durable execution engine that survives crashes and long waits. - Use when: Work spanning minutes or hours that must survive restarts and retries. - Skip when: You need it live in a day — the adoption cost is real. - Substitutes: Inngest (Workflows), Trigger.dev (Workflows), Restate (Workflows), DBOS (Workflows) - [Inngest](https://lattice.kunjshah.in/workflow-orchestration/inngest) — Event-driven step functions with durable replay for TypeScript. - Use when: Durable step functions in TypeScript, without running the engine. - Skip when: You are not working in TypeScript. - Substitutes: Temporal (Workflows), Trigger.dev (Workflows) - [Trigger.dev](https://lattice.kunjshah.in/workflow-orchestration/trigger-dev) — Background jobs and AI workflows with long-running task support. - Use when: Long-running background jobs that must survive a deploy. - Skip when: You need more than TypeScript. - Substitutes: Inngest (Workflows), Fly Machines (Workflows), Temporal (Workflows) - [Dagster](https://lattice.kunjshah.in/workflow-orchestration/dagster) — Asset-oriented orchestration for data and ML pipelines. - Use when: Data and ML pipelines framed as assets with explicit dependencies. - Skip when: You want a task queue rather than a lineage-aware orchestrator. - Substitutes: Prefect (Workflows), Apache Airflow (Workflows) - [Prefect](https://lattice.kunjshah.in/workflow-orchestration/prefect) — Python-native workflow orchestration with a managed cloud option. - Use when: Python-native flows with a managed cloud option behind them. - Skip when: You need deep lineage modelling. - Substitutes: Dagster (Workflows), Apache Airflow (Workflows) - [Apache Airflow](https://lattice.kunjshah.in/workflow-orchestration/apache-airflow) — Scheduler and DAG engine long used for batch data engineering. - Use when: Scheduled batch data engineering, where DAGs are the shared language. - Skip when: You are building a latency-sensitive product — it was not designed for one. - Substitutes: Dagster (Workflows), Prefect (Workflows) - [Restate](https://lattice.kunjshah.in/workflow-orchestration/restate) — Durable execution with a low-latency stateful API surface. - Use when: Durable execution with a genuinely low-latency stateful API surface. - Skip when: You need the largest ecosystem around the engine. - Substitutes: Temporal (Workflows), DBOS (Workflows) - [DBOS](https://lattice.kunjshah.in/workflow-orchestration/dbos) — Durable workflows as ordinary Python functions and decorators. - Use when: Durable workflows as ordinary Python functions and decorators. - Skip when: You want a separate service to operate. - Substitutes: Temporal (Workflows), Restate (Workflows) - [Fly Machines](https://lattice.kunjshah.in/workflow-orchestration/fly-machines) — Hosting for stateful containers that fits durable agent workers. - Use when: Hosting stateful containers that fits durable agent workers. - Skip when: You want to own the infrastructure layer. - Substitutes: Modal (Workflows), Trigger.dev (Workflows) - [Modal](https://lattice.kunjshah.in/workflow-orchestration/modal) — Serverless GPU and container platform popular for batch inference. - Use when: Serverless GPUs and containers for batch inference and scheduled jobs. - Skip when: Data locality rules forbid ephemeral compute. - Substitutes: Fly Machines (Workflows), Replicate (Training) ## [Guardrails & Safety](https://lattice.kunjshah.in/guardrails-safety) Stops the bad input before it, and the bad output after. - [NeMo Guardrails](https://lattice.kunjshah.in/guardrails-safety/nemo-guardrails) — Programmable rails that constrain conversational flow. - Use when: Constraining conversational flow with programmable rails. - Skip when: You need deep semantic moderation — it is not a classifier. - Substitutes: Guardrails AI (Guardrails), Llama Guard (Guardrails) - [Guardrails AI](https://lattice.kunjshah.in/guardrails-safety/guardrails-ai) — Validators that check model output against a defined schema. - Use when: Validators that check output against a schema you define. - Skip when: You need free-form moderation rather than structural checks. - Substitutes: NeMo Guardrails (Guardrails), Invariant Guardrails (Guardrails) - [Llama Guard](https://lattice.kunjshah.in/guardrails-safety/llama-guard) — Safety classifier for prompt and response moderation. - Use when: One classifier covering both prompt and response moderation. - Skip when: You need domain-specific policy enforcement. - Substitutes: NeMo Guardrails (Guardrails), Lakera Guard (Guardrails) - [Microsoft Presidio](https://lattice.kunjshah.in/guardrails-safety/microsoft-presidio) — PII detection and anonymization for text and images. - [Lakera Guard](https://lattice.kunjshah.in/guardrails-safety/lakera-guard) — Prompt injection and jailbreak detection at the gateway. - Use when: Prompt-injection and jailbreak detection at the gateway. - Skip when: Requests cannot leave your network. - Substitutes: Llama Guard (Guardrails), Invariant Guardrails (Guardrails) - [garak](https://lattice.kunjshah.in/guardrails-safety/garak) — Scanner that probes models for known vulnerability classes. - Use when: Probing your own system for known vulnerability classes. - Skip when: You have no remediation path for whatever it finds. - Substitutes: PyRIT (Guardrails), promptfoo (Evals) - [PyRIT](https://lattice.kunjshah.in/guardrails-safety/pyrit) — Microsoft's red-team tool for generating and scoring attack prompts. - Use when: Generating and scoring attack prompts systematically. - Skip when: You are not authorised to test the target. - Substitutes: garak (Guardrails) - [Invariant Guardrails](https://lattice.kunjshah.in/guardrails-safety/invariant-guardrails) — Guardrails as code, enforced inline in the call path. - Use when: Guardrails as code, enforced inline in the call path. - Skip when: Self-hosting is a hard requirement. - Substitutes: Guardrails AI (Guardrails), Lakera Guard (Guardrails) ## [Prompt Engineering](https://lattice.kunjshah.in/prompt-engineering) Makes the prompt an artifact you can diff and test. - [DSPy](https://lattice.kunjshah.in/prompt-engineering/dspy) — Declarative prompting that compiles to optimized programs from examples. - Use when: Compiling prompts into optimised programs from labelled examples. - Skip when: You have no examples and no way to score them. - Substitutes: TextGrad (Prompts), PromptLayer (Prompts) - [Instructor](https://lattice.kunjshah.in/prompt-engineering/instructor) — Schema-constrained extraction with automatic validation and retry. - Use when: Schema-validated extraction with typed retry and partial streaming. - Skip when: You need a hard token-level guarantee rather than validation. - Substitutes: Guidance (Prompts), Outlines (Prompts) - [Humanloop](https://lattice.kunjshah.in/prompt-engineering/humanloop) — Prompt versioning and evaluation for production teams. - Use when: Prompt versioning and evaluation run by a product team. - Skip when: Prompts or data cannot be sent to a vendor. - Substitutes: PromptLayer (Prompts), Agenta (Prompts), Promptwatch (Prompts), Vellum (Evals) - [Guidance](https://lattice.kunjshah.in/prompt-engineering/guidance) — Constrained generation by interleaving control flow and model output. - Use when: Constrained decoding with control flow interleaved into generation. - Skip when: You need better ergonomics for plain extraction. - Substitutes: Instructor (Prompts), Outlines (Prompts) - [PromptLayer](https://lattice.kunjshah.in/prompt-engineering/promptlayer) — Prompt registry with request logging and regression testing. - Use when: A prompt registry with request logging and regression tests. - Skip when: You need self-hosting. - Substitutes: DSPy (Prompts), Humanloop (Prompts), Agenta (Prompts), Promptwatch (Prompts) - [Agenta](https://lattice.kunjshah.in/prompt-engineering/agenta) — Open-source prompt management with a playground and versioning. - Use when: Open-source prompt management with a playground and versioning. - Skip when: You need eval depth rather than prompt ergonomics. - Substitutes: Humanloop (Prompts), PromptLayer (Prompts), Promptwatch (Prompts) - [Promptwatch](https://lattice.kunjshah.in/prompt-engineering/promptwatch) — Prompt regression tests and side-by-side comparison. - Use when: Prompt regression tests and side-by-side comparison. - Skip when: You need self-hosting. - Substitutes: PromptLayer (Prompts), Agenta (Prompts), Humanloop (Prompts) - [TextGrad](https://lattice.kunjshah.in/prompt-engineering/textgrad) — Automatic prompt optimisation via textual feedback loops. - Use when: Automatic prompt optimisation via textual feedback loops. - Skip when: Optimising without an eval set is guessing. - Substitutes: DSPy (Prompts) - [Outlines](https://lattice.kunjshah.in/prompt-engineering/outlines) — Constrained generation that masks the token space to valid output. - Use when: Token-level constrained generation from a JSON schema or grammar. - Skip when: Post-hoc validation is enough for your use case. - Substitutes: Instructor (Prompts), Guidance (Prompts) ## [Evaluation & Observability](https://lattice.kunjshah.in/evaluation-observability) The only way to know whether any of the above works. - [LangSmith](https://lattice.kunjshah.in/evaluation-observability/langsmith) — Tracing, evaluation and dataset tooling across LangChain runs. - Use when: Teams already deep in LangChain needing tracing and datasets. - Skip when: Traces cannot leave your infrastructure. - Substitutes: Braintrust (Evals), Langfuse (Evals), Weights & Biases (Evals), Arize Phoenix (Evals) - [Braintrust](https://lattice.kunjshah.in/evaluation-observability/braintrust) — Evaluation platform with built-in scorers and a data flywheel. - Use when: Making evaluation, rather than tracing, the primary workflow. - Skip when: You need open formats and self-hosting. - Substitutes: TrueFoundry (Routing), LangSmith (Evals), Langfuse (Evals), DeepEval (Evals), Vellum (Evals) - [Arize Phoenix](https://lattice.kunjshah.in/evaluation-observability/arize-phoenix) — Open-source tracing and evaluation built on OpenTelemetry. - Use when: OpenTelemetry-native evaluation you can run yourself. - Skip when: You want a vendor support contract behind it. - Substitutes: LangSmith (Evals), Langfuse (Evals), Gentrace (Evals), Opik (Evals) - [Langfuse](https://lattice.kunjshah.in/evaluation-observability/langfuse) — Self-hostable LLM tracing, prompt management and cost analytics. - Use when: Self-hostable tracing, prompts and evals in one place. - Skip when: You want a fully managed product with a support contract. - Substitutes: LangSmith (Evals), Braintrust (Evals), Arize Phoenix (Evals), Helicone (Evals), Gentrace (Evals), Opik (Evals) - [promptfoo](https://lattice.kunjshah.in/evaluation-observability/promptfoo) — Declarative red-teaming and regression tests for prompts and agents. - Use when: Declarative red-teaming and regression tests that run in CI. - Skip when: You need a managed UI rather than a test runner. - Substitutes: DeepEval (Evals), Evidently AI (Evals), garak (Guardrails) - [DeepEval](https://lattice.kunjshah.in/evaluation-observability/deepeval) — Open-source pytest-style evaluation suite for LLM outputs. - Use when: pytest-style evaluation you can run next to your unit tests. - Skip when: You need a platform rather than a library. - Substitutes: Braintrust (Evals), promptfoo (Evals), Evidently AI (Evals) - [Helicone](https://lattice.kunjshah.in/evaluation-observability/helicone) — Gateway-level observability with per-request cost and latency. - Use when: Gateway-level observability with per-request cost and latency. - Skip when: You need eval primitives more than traffic logs. - Substitutes: Gentrace (Evals), Langfuse (Evals) - [Weights & Biases](https://lattice.kunjshah.in/evaluation-observability/weights-biases) — Experiment tracking and model registry with LLM eval surfaces. - Use when: Experiment tracking and a model registry, with eval surfaces on top. - Skip when: You need an open, exportable format. - Substitutes: Weights & Biases Launch (Training), LangSmith (Evals) - [OpenTelemetry](https://lattice.kunjshah.in/evaluation-observability/opentelemetry) — Vendor-neutral standard for emitting traces and metrics. - [Vellum](https://lattice.kunjshah.in/evaluation-observability/vellum) — Prompt and eval platform with a visual debugger for chains. - Use when: Prompt and eval work with a visual debugger for chains. - Skip when: Self-hosting or reproducibility is a requirement. - Substitutes: Humanloop (Prompts), Braintrust (Evals) - [Gentrace](https://lattice.kunjshah.in/evaluation-observability/gentrace) — Open-source tracing and dashboards for LLM applications. - Use when: Open-source tracing and dashboards for LLM applications. - Skip when: You want eval depth more than traffic visibility. - Substitutes: Arize Phoenix (Evals), Langfuse (Evals), Helicone (Evals), Opik (Evals) - [Opik](https://lattice.kunjshah.in/evaluation-observability/opik) — Open-source LLM observability and evaluation from Comet. - Use when: Open-source observability and evaluation you run yourself. - Skip when: You need enterprise support behind it. - Substitutes: Arize Phoenix (Evals), Langfuse (Evals), Gentrace (Evals) - [Evidently AI](https://lattice.kunjshah.in/evaluation-observability/evidently-ai) — Evaluation and monitoring for both classical and LLM systems. - Use when: One evaluation model for both classical ML and LLM systems. - Skip when: You only need LLM-specific tracing. - Substitutes: DeepEval (Evals), promptfoo (Evals) ## [Learning & Reference](https://lattice.kunjshah.in/learning-reference) Not a layer. Reading, once you know what to look for. - [Hugging Face](https://lattice.kunjshah.in/learning-reference/hugging-face) — Model hub, datasets and open-source library ecosystem. - [Full Stack Deep Learning](https://lattice.kunjshah.in/learning-reference/full-stack-deep-learning) — Course covering the practical side of shipping LLM systems. - [Sebastian Raschka](https://lattice.kunjshah.in/learning-reference/sebastian-raschka) — Essays and books distilling research into working knowledge. - [arXiv](https://lattice.kunjshah.in/learning-reference/arxiv) — Primary preprint archive for machine learning research. - [Papers with Code](https://lattice.kunjshah.in/learning-reference/papers-with-code) — Links papers to their reference implementations and benchmarks. - [Jay Alammar](https://lattice.kunjshah.in/learning-reference/jay-alammar) — Visual explanations of transformers, RAG and LLM internals. - [AI Engineer](https://lattice.kunjshah.in/learning-reference/ai-engineer) — Conference and community focused on shipping LLM products. - [Hugging Face Cookbook](https://lattice.kunjshah.in/learning-reference/hugging-face-cookbook) — Task-oriented notebooks for fine-tuning, serving and RAG. - [Made With ML](https://lattice.kunjshah.in/learning-reference/made-with-ml) — Course on designing, building and deploying ML systems. - [Distill](https://lattice.kunjshah.in/learning-reference/distill) — Archive of carefully explained model and method explainers.