The infrastructure behind working AI systems.

Turns weights into throughput.

• vLLMruntimePaged-attention inference engine with an OpenAI-compatible server.vllm.aiMore about vLLM in this index
• SGLangruntimeStructured generation runtime built on RadixAttention prefix reuse.github.comMore about SGLang in this index
• llama.cppruntimePortable CPU/GPU inference for quantized GGUF models.github.comMore about llama.cpp in this index
• OllamaruntimeLocal model runner with a single-binary distribution and HTTP API.ollama.comMore about Ollama in this index
• TensorRT-LLMruntimeNVIDIA-optimized inference with inflight batching and FP8.github.comMore about TensorRT-LLM in this index
• Text Generation InferenceruntimeHugging Face production server for LLMs with tensor parallelism.github.comMore about Text Generation Inference in this index
• LM StudioserviceDesktop app for running and serving local models with an OpenAI API.lmstudio.aiMore about LM Studio in this index
• Triton Inference ServerruntimeMulti-framework inference server for ONNX, TensorRT and Python.github.comMore about Triton Inference Server in this index
• llamafileruntimePackages a model and its runtime into a single executable.github.comMore about llamafile in this index
• KoboldCppruntimeGGUF inference with a focused text-adventure UI and API.github.comMore about KoboldCpp in this index
• PowerInferruntimeCPU-first runtime that offloads hot paths to the GPU.github.comMore about PowerInfer in this index
• MarlinlibraryQuantised GEMM kernels for near-GPU speed at 4-bit.github.comMore about Marlin in this index

Decides which model answers, and what it costs.

• LiteLLMlibraryOpenAI-format proxy translating across 100+ model providers.litellm.aiMore about LiteLLM in this index
• PortkeyplatformAI gateway with routing, caching and guardrails in one layer.portkey.aiMore about Portkey in this index
• Cloudflare AI GatewayserviceEdge gateway adding caching, retries and rate limiting.developers.cloudflare.comMore about Cloudflare AI Gateway in this index
• OpenRouterserviceSingle API key and billing across many hosted models.openrouter.aiMore about OpenRouter in this index
• MartianplatformModel router that optimizes for quality, cost and latency.withmartian.comMore about Martian in this index
• Envoy AI GatewayruntimeCNCF-track gateway for LLM traffic on an Envoy data plane.aigateway.envoyproxy.ioMore about Envoy AI Gateway in this index
• BifrostlibraryHigh-throughput LLM gateway with drop-in OpenAI compatibility.getmaxim.aiMore about Bifrost in this index
• RouteLLMlibraryLearned router that cuts cost by matching difficulty to model size.github.comMore about RouteLLM in this index
• Not DiamondplatformRouting and prompt optimisation tuned on your own traffic.notdiamond.aiMore about Not Diamond in this index
• TrueFoundryplatformGateway plus observability and guardrails as one deployment.truefoundry.comMore about TrueFoundry in this index

Holds the embeddings, returns the few that matter.

• pgvectordatabaseVector similarity search as a Postgres extension.github.comMore about pgvector in this index
• QdrantdatabaseRust vector database with rich filtering payloads.qdrant.techMore about Qdrant in this index
• WeaviatedatabaseGraph-aware vector database with hybrid and multi-vector search.weaviate.ioMore about Weaviate in this index
• ChromadatabaseEmbeddings database designed for fast prototyping.trychroma.comMore about Chroma in this index
• PineconedatabaseFully managed vector database with serverless scaling.pinecone.ioMore about Pinecone in this index
• MilvusdatabaseCloud-native vector database supporting billion-scale indexes.milvus.ioMore about Milvus in this index
• TurbopufferdatabaseVector search engine tuned for high-recall retrieval workloads.turbopuffer.comMore about Turbopuffer in this index
• UnstructuredlibraryPreprocessing library that partitions raw documents for indexing.unstructured.ioMore about Unstructured in this index
• ElasticsearchdatabaseBM25 plus dense vectors in one index, with hybrid ranking.elastic.coMore about Elasticsearch in this index
• VespaplatformSearch engine built for ranking-heavy retrieval at scale.vespa.aiMore about Vespa in this index
• RerankersservicePurpose-built cross-encoders for the second retrieval stage.cohere.comMore about Rerankers in this index
• Jina AIserviceMultimodal embeddings and a reranker behind one endpoint.jina.aiMore about Jina AI in this index
• LangChain Text SplitterslibraryDocument loaders and splitters for the ingest stage.python.langchain.comMore about LangChain Text Splitters in this index
• DoclinglibraryLayout-aware parsing that preserves table and heading structure.github.comMore about Docling in this index
• LlamaParseserviceManaged parsing for PDFs, especially the ugly ones.llamaindex.aiMore about LlamaParse in this index

Produces the weights that layer 1 serves.

• UnslothlibraryHand-written kernels that cut LoRA memory and time sharply.unsloth.aiMore about Unsloth in this index
• AxolotllibraryConfiguration-driven fine-tuning across common architectures.axolotl.aiMore about Axolotl in this index
• LLaMA-FactoryplatformUnified interface for SFT, DPO and RLHF on open models.github.comMore about LLaMA-Factory in this index
• PEFTlibraryParameter-efficient fine-tuning methods such as LoRA and QLoRA.github.comMore about PEFT in this index
• TRLlibraryHugging Face library of post-training trainers for SFT, DPO and GRPO.github.comMore about TRL in this index
• DeepSpeedlibraryZeRO sharding and pipeline parallelism for large-scale training.deepspeed.aiMore about DeepSpeed in this index
• Megatron-LMlibraryTensor and pipeline parallel training for multi-GPU clusters.github.comMore about Megatron-LM in this index
• torchtunelibraryPyTorch-native recipes for fine-tuning and aligning open models.github.comMore about torchtune in this index
• Hugging Face TRLplatformPreference and reward optimisation on top of any PEFT setup.huggingface.coMore about Hugging Face TRL in this index
• ColabplatformHosted GPUs for small runs and quick experiments.colab.research.google.comMore about Colab in this index
• ReplicateplatformHosted fine-tuning and deployment for open models.replicate.comMore about Replicate in this index
• Weights & Biases LaunchplatformManaged training runs with sweeps and artifact tracking.wandb.aiMore about Weights & Biases Launch in this index

Turns a model call into a multi-step program.

• LangChainframeworkComposable abstractions for model calls, tools and state.langchain.comMore about LangChain in this index
• LlamaIndexframeworkData-centric framework for retrieval and agent workflows.llamaindex.aiMore about LlamaIndex in this index
• Pydantic AIframeworkType-safe agent framework that leans on Pydantic models.ai.pydantic.devMore about Pydantic AI in this index
• AutoGenframeworkMicrosoft research project for conversable multi-agent systems.microsoft.github.ioMore about AutoGen in this index
• CrewAIframeworkRole-based orchestration where agents collaborate as a crew.crewai.comMore about CrewAI in this index
• Semantic KernelframeworkMicrosoft SDK for embedding AI steps into .NET and Python apps.learn.microsoft.comMore about Semantic Kernel in this index
• MastraframeworkTypeScript agent framework with typed workflows and evals.mastra.aiMore about Mastra in this index
• OpenAI Agents SDKframeworkLightweight primitives for handoffs, guardrails and tracing.openai.github.ioMore about OpenAI Agents SDK in this index
• AgnoframeworkMinimal agent runtime centered on model-agnostic tool interfaces.agno.comMore about Agno in this index
• Claude Agent SDKframeworkAnthropic's toolkit for building agents with tool use and hooks.docs.anthropic.comMore about Claude Agent SDK in this index
• LettaplatformAgent runtime with persistent, editable memory as a first-class primitive.letta.comMore about Letta in this index
• smolagentsframeworkMinimal code-first agent loop from the Hugging Face team.huggingface.coMore about smolagents in this index
• Vercel AI SDKlibraryProvider-agnostic building blocks for streaming apps and agents.ai-sdk.devMore about Vercel AI SDK in this index

Survives the retries, the waits and the crashes.

• TemporalplatformDurable execution engine that survives crashes and long waits.temporal.ioMore about Temporal in this index
• InngestplatformEvent-driven step functions with durable replay for TypeScript.inngest.comMore about Inngest in this index
• Trigger.devplatformBackground jobs and AI workflows with long-running task support.trigger.devMore about Trigger.dev in this index
• DagsterplatformAsset-oriented orchestration for data and ML pipelines.dagster.ioMore about Dagster in this index
• PrefectplatformPython-native workflow orchestration with a managed cloud option.prefect.ioMore about Prefect in this index
• Apache AirflowplatformScheduler and DAG engine long used for batch data engineering.apache.orgMore about Apache Airflow in this index
• RestateplatformDurable execution with a low-latency stateful API surface.restate.devMore about Restate in this index
• DBOSlibraryDurable workflows as ordinary Python functions and decorators.dbos.devMore about DBOS in this index
• Fly MachinesplatformHosting for stateful containers that fits durable agent workers.fly.ioMore about Fly Machines in this index
• ModalplatformServerless GPU and container platform popular for batch inference.modal.comMore about Modal in this index

Stops the bad input before it, and the bad output after.

• NeMo GuardrailslibraryProgrammable rails that constrain conversational flow.github.comMore about NeMo Guardrails in this index
• Guardrails AIlibraryValidators that check model output against a defined schema.guardrailsai.comMore about Guardrails AI in this index
• Llama GuardlibrarySafety classifier for prompt and response moderation.ai.meta.comMore about Llama Guard in this index
• Microsoft PresidiolibraryPII detection and anonymization for text and images.microsoft.github.ioMore about Microsoft Presidio in this index
• Lakera GuardservicePrompt injection and jailbreak detection at the gateway.lakera.aiMore about Lakera Guard in this index
• garaklibraryScanner that probes models for known vulnerability classes.github.comMore about garak in this index
• PyRITlibraryMicrosoft's red-team tool for generating and scoring attack prompts.github.comMore about PyRIT in this index
• Invariant GuardrailsplatformGuardrails as code, enforced inline in the call path.invariantlabs.aiMore about Invariant Guardrails in this index

Makes the prompt an artifact you can diff and test.

• DSPylibraryDeclarative prompting that compiles to optimized programs from examples.dspy.aiMore about DSPy in this index
• InstructorlibrarySchema-constrained extraction with automatic validation and retry.python.useinstructor.comMore about Instructor in this index
• HumanloopplatformPrompt versioning and evaluation for production teams.humanloop.comMore about Humanloop in this index
• GuidancelibraryConstrained generation by interleaving control flow and model output.guidance.mit.eduMore about Guidance in this index
• PromptLayerplatformPrompt registry with request logging and regression testing.promptlayer.comMore about PromptLayer in this index
• AgentaplatformOpen-source prompt management with a playground and versioning.agenta.aiMore about Agenta in this index
• PromptwatchplatformPrompt regression tests and side-by-side comparison.promptwatch.comMore about Promptwatch in this index
• TextGradlibraryAutomatic prompt optimisation via textual feedback loops.github.comMore about TextGrad in this index
• OutlineslibraryConstrained generation that masks the token space to valid output.dottxt-ai.github.ioMore about Outlines in this index

The only way to know whether any of the above works.

• LangSmithplatformTracing, evaluation and dataset tooling across LangChain runs.smith.langchain.comMore about LangSmith in this index
• BraintrustplatformEvaluation platform with built-in scorers and a data flywheel.braintrust.devMore about Braintrust in this index
• Arize PhoenixlibraryOpen-source tracing and evaluation built on OpenTelemetry.phoenix.arize.comMore about Arize Phoenix in this index
• LangfuseplatformSelf-hostable LLM tracing, prompt management and cost analytics.langfuse.comMore about Langfuse in this index
• promptfoolibraryDeclarative red-teaming and regression tests for prompts and agents.promptfoo.devMore about promptfoo in this index
• DeepEvallibraryOpen-source pytest-style evaluation suite for LLM outputs.deepeval.comMore about DeepEval in this index
• HeliconeplatformGateway-level observability with per-request cost and latency.helicone.aiMore about Helicone in this index
• Weights & BiasesplatformExperiment tracking and model registry with LLM eval surfaces.wandb.aiMore about Weights & Biases in this index
• OpenTelemetrylibraryVendor-neutral standard for emitting traces and metrics.opentelemetry.ioMore about OpenTelemetry in this index
• VellumplatformPrompt and eval platform with a visual debugger for chains.vellum.aiMore about Vellum in this index
• GentraceplatformOpen-source tracing and dashboards for LLM applications.gentrace.aiMore about Gentrace in this index
• OpikplatformOpen-source LLM observability and evaluation from Comet.comet.comMore about Opik in this index
• Evidently AIlibraryEvaluation and monitoring for both classical and LLM systems.evidentlyai.comMore about Evidently AI in this index

Not a layer. Reading, once you know what to look for.

• Hugging FaceplatformModel hub, datasets and open-source library ecosystem.huggingface.coMore about Hugging Face in this index
• Full Stack Deep LearningreadingCourse covering the practical side of shipping LLM systems.fullstackdeeplearning.comMore about Full Stack Deep Learning in this index
• Sebastian RaschkareadingEssays and books distilling research into working knowledge.magazine.sebastianraschka.comMore about Sebastian Raschka in this index
• arXivreadingPrimary preprint archive for machine learning research.arxiv.orgMore about arXiv in this index
• Papers with CodereadingLinks papers to their reference implementations and benchmarks.paperswithcode.comMore about Papers with Code in this index
• Jay AlammarreadingVisual explanations of transformers, RAG and LLM internals.jalammar.github.ioMore about Jay Alammar in this index
• AI EngineerreadingConference and community focused on shipping LLM products.ai.engineerMore about AI Engineer in this index
• Hugging Face CookbookreadingTask-oriented notebooks for fine-tuning, serving and RAG.huggingface.coMore about Hugging Face Cookbook in this index
• Made With MLreadingCourse on designing, building and deploying ML systems.madewithml.comMore about Made With ML in this index
• DistillreadingArchive of carefully explained model and method explainers.distill.pubMore about Distill in this index