Lattice
Skip to content

112 tools · 73 self-hosted · 68 permissively licensed

Every AI infrastructure tool, filterable.

The whole index in one list. Use it when you know what you are looking for; use the stack diagram when you do not.

112
Section
Role
Deployment
Kind
Cost
  • vLLMruntimeApache-2.0· InferencePaged-attention inference engine with an OpenAI-compatible server.Use when General GPU serving under bursty, high-concurrency traffic.Skip when Traffic is steady and low-concurrency, so batches never fill.
  • SGLangruntimeApache-2.0· InferenceStructured generation runtime built on RadixAttention prefix reuse.Use when A large share of requests share a long prefix, or you need constrained decoding.Skip when You want the fewest moving parts and vLLM already clears your bar.
  • llama.cppruntimeMIT· InferencePortable CPU/GPU inference for quantized GGUF models.Use when The constraint is hardware, not throughput — no GPU, or an edge box.Skip when You need maximum concurrent throughput on datacentre GPUs.
  • OllamaruntimeMIT· InferenceLocal model runner with a single-binary distribution and HTTP API.Use when Local model access for a developer or small team, with no setup.Skip when You need production throughput or a stable server API.
  • TensorRT-LLMruntimeApache-2.0· InferenceNVIDIA-optimized inference with inflight batching and FP8.Use when You are committed to NVIDIA hardware and need the last of the throughput.Skip when Hardware portability matters, or you have no GPUs to tune against.
  • Text Generation InferenceruntimeApache-2.0· InferenceHugging Face production server for LLMs with tensor parallelism.Use when Multi-GPU tensor parallelism inside an existing Hugging Face stack.Skip when The model fits on one GPU, where this is overkill.
  • LM Studioserviceproprietary· InferenceDesktop app for running and serving local models with an OpenAI API.Use when A desktop GUI for running and serving local models, aimed at one operator.Skip when You need headless, multi-tenant deployment.
  • Triton Inference ServerruntimeBSD-3-Clause· InferenceMulti-framework inference server for ONNX, TensorRT and Python.Use when One endpoint serving ONNX, TensorRT and Python models side by side.Skip when You serve one model family and want peak LLM throughput.
  • llamafileruntimeApache-2.0· InferencePackages a model and its runtime into a single executable.Use when Shipping a model as a single executable with no install step.Skip when You need batching, multi-GPU serving or a production HTTP server.
  • KoboldCppruntimeMIT· InferenceGGUF inference with a focused text-adventure UI and API.Use when Local generation with a UI tuned for interactive, long-form use.Skip when You need headless serving at scale.
  • PowerInferruntimeApache-2.0· InferenceCPU-first runtime that offloads hot paths to the GPU.Use when CPU-first serving where the GPU accelerates rather than carries the load.Skip when You have GPUs to spare — it trades peak throughput for reach.
  • MarlinlibraryApache-2.0· InferenceQuantised GEMM kernels for near-GPU speed at 4-bit.Use when Near-GPU throughput from 4-bit weights, as a kernel inside another runtime.Skip when You need a scheduler and a server as well as a kernel.
  • LiteLLMlibraryMIT· RoutingOpenAI-format proxy translating across 100+ model providers.Use when The widest provider coverage, deployed the least committal way.Skip when You want routing policy to live in a SaaS console.
  • Portkeyplatformproprietary· RoutingAI gateway with routing, caching and guardrails in one layer.Use when A managed gateway with guardrails and analytics already bundled.Skip when Prompts cannot leave your infrastructure.also in guardrails-safety
  • Cloudflare AI Gatewayserviceproprietary· RoutingEdge gateway adding caching, retries and rate limiting.Use when Edge caching and latency dominate, on Cloudflare already.Skip when Requests must not transit a third party's network.
  • OpenRouterserviceproprietary· RoutingSingle API key and billing across many hosted models.Use when One key and one bill across many hosted models, while experimenting.Skip when You need self-hosting or per-provider SLAs.
  • Martianplatformproprietary· RoutingModel router that optimizes for quality, cost and latency.Use when Routing tuned on your own traffic rather than a public benchmark.Skip when You want the routing rule to be plain config you can read.
  • Envoy AI GatewayruntimeApache-2.0· RoutingCNCF-track gateway for LLM traffic on an Envoy data plane.Use when A Kubernetes shop that already runs Envoy and needs the gateway in-cluster.Skip when You want a gateway configurable without a data plane.
  • Bifrostlibraryproprietary· RoutingHigh-throughput LLM gateway with drop-in OpenAI compatibility.Use when A high-throughput OpenAI-compatible proxy with a small footprint.Skip when You need the breadth of a full routing stack.
  • RouteLLMlibraryApache-2.0· RoutingLearned router that cuts cost by matching difficulty to model size.Use when Cutting cost by matching query difficulty to model size, with a learned router.Skip when You cannot evaluate the quality you would be trading away.
  • Not Diamondplatformproprietary· RoutingRouting and prompt optimisation tuned on your own traffic.Use when Routing and prompt optimisation tuned on your own traffic.Skip when Self-hosting or auditability is a requirement.
  • TrueFoundryplatformproprietary· RoutingGateway plus observability and guardrails as one deployment.Use when Gateway, observability and guardrails as one managed deployment.Skip when You want to adopt the pieces independently.also in evaluation-observability, guardrails-safety
  • pgvectordatabasePostgreSQL· RetrievalVector similarity search as a Postgres extension.Use when Under a few million vectors, or when retrieval joins rows you already have.Skip when Vector search has become the workload rather than a side feature.
  • QdrantdatabaseApache-2.0· RetrievalRust vector database with rich filtering payloads.Use when Vector-first workloads with heavy metadata filtering at scale.Skip when You want no new datastore to operate.
  • WeaviatedatabaseBSD-3-Clause· RetrievalGraph-aware vector database with hybrid and multi-vector search.Use when Hybrid or multi-vector search with a graph-shaped data model.Skip when Your data is genuinely relational and joins matter more than vectors.
  • ChromadatabaseApache-2.0· RetrievalEmbeddings database designed for fast prototyping.Use when Getting a prototype working in an afternoon.Skip when You need durability, scaling or concurrent writes.
  • Pineconedatabaseproprietary· RetrievalFully managed vector database with serverless scaling.Use when Zero operational work, and vector search is not your core competence.Skip when Residency, cost predictability or a query language you control.
  • MilvusdatabaseApache-2.0· RetrievalCloud-native vector database supporting billion-scale indexes.Use when Billion-scale indexes with a cloud-native deployment.Skip when You need a small, comprehensible system.
  • Turbopufferdatabaseproprietary· RetrievalVector search engine tuned for high-recall retrieval workloads.Use when High-recall retrieval at volume, without operating the index yourself.Skip when You need to tune the index directly.
  • UnstructuredlibraryApache-2.0· RetrievalPreprocessing library that partitions raw documents for indexing.Use when Partitioning messy documents into retrievable chunks before indexing.Skip when Your input is already structured.
  • ElasticsearchdatabaseElastic-License-2.0· RetrievalBM25 plus dense vectors in one index, with hybrid ranking.Use when One index carrying BM25 and dense vectors, with ranking you can tune.Skip when You need a permissive licence — the Elastic licence is not OSI-approved.
  • VespaplatformApache-2.0· RetrievalSearch engine built for ranking-heavy retrieval at scale.Use when Ranking-heavy retrieval where the ranking function is the product.Skip when You want a small, low-operations component.
  • Rerankersserviceproprietary· RetrievalPurpose-built cross-encoders for the second retrieval stage.Use when A cross-encoder in the second retrieval stage, to lift precision cheaply.Skip when You have no way to measure whether recall actually improved.
  • Jina AIserviceproprietary· RetrievalMultimodal embeddings and a reranker behind one endpoint.Use when Multimodal embeddings and a reranker behind one endpoint.Skip when Embeddings must stay in-house.
  • LangChain Text SplitterslibraryMIT· RetrievalDocument loaders and splitters for the ingest stage.Use when Document loaders and splitters, if you are already in that ecosystem.Skip when Structure matters — fixed-size splitting is rarely the right boundary.
  • DoclinglibraryMIT· RetrievalLayout-aware parsing that preserves table and heading structure.Use when Layout-aware parsing that preserves tables and heading structure.Skip when Your documents are plain text.
  • LlamaParseserviceproprietary· RetrievalManaged parsing for PDFs, especially the ugly ones.Use when Managed parsing for PDFs, especially the ugly ones.Skip when Document contents cannot leave your infrastructure.
  • UnslothlibraryApache-2.0· TrainingHand-written kernels that cut LoRA memory and time sharply.Use when LoRA fine-tuning on a single GPU, with memory and time roughly halved.Skip when You need a training stack you can audit line by line.
  • AxolotllibraryApache-2.0· TrainingConfiguration-driven fine-tuning across common architectures.Use when Configuration-driven fine-tuning across many architectures.Skip when You need to hand-tune the training loop itself.
  • LLaMA-FactoryplatformApache-2.0· TrainingUnified interface for SFT, DPO and RLHF on open models.Use when One interface for SFT, DPO and RLHF across open models.Skip when You want a minimal, readable training script.
  • PEFTlibraryBSD-3-Clause· TrainingParameter-efficient fine-tuning methods such as LoRA and QLoRA.Use when Parameter-efficient fine-tuning — LoRA and variants — on limited hardware.Skip when You need to change the model's capabilities, not bolt on an adapter.
  • TRLlibraryApache-2.0· TrainingHugging Face library of post-training trainers for SFT, DPO and GRPO.Use when Post-training trainers for SFT, DPO and GRPO on top of any PEFT setup.Skip when You are training from scratch rather than aligning an existing model.
  • DeepSpeedlibraryMIT· TrainingZeRO sharding and pipeline parallelism for large-scale training.Use when ZeRO sharding and pipeline parallelism across many GPUs.Skip when The model fits on one device — the complexity buys nothing.
  • Megatron-LMlibraryApache-2.0· TrainingTensor and pipeline parallel training for multi-GPU clusters.Use when Tensor and pipeline parallel pretraining at cluster scale.Skip when You are fine-tuning rather than pretraining.
  • torchtunelibraryBSD-3-Clause· TrainingPyTorch-native recipes for fine-tuning and aligning open models.Use when PyTorch-native recipes that you can read and modify.Skip when You want the breadth of a config-driven stack.
  • Hugging Face TRLplatformproprietary· TrainingPreference and reward optimisation on top of any PEFT setup.Use when Preference and reward optimisation hosted, without a cluster.Skip when Training data cannot be uploaded to a third party.
  • Colabplatformproprietary· TrainingHosted GPUs for small runs and quick experiments.Use when A disposable GPU for a quick experiment, with no setup.Skip when Anything reproducible — a notebook is not a training pipeline.
  • Replicateplatformproprietary· TrainingHosted fine-tuning and deployment for open models.Use when Hosted fine-tuning and deployment of an open model, billed per run.Skip when You need custom training code or strict residency.
  • Weights & Biases Launchplatformproprietary· TrainingManaged training runs with sweeps and artifact tracking.Use when Managed training runs with sweeps and artifact tracking wired up.Skip when You want training infrastructure you run yourself.also in evaluation-observability
  • LangChainframeworkMIT· AgentsComposable abstractions for model calls, tools and state.Use when Tool use and state, in an ecosystem where the answers already exist.Skip when You want a thin, legible core — this is a large surface.
  • LlamaIndexframeworkMIT· AgentsData-centric framework for retrieval and agent workflows.Use when Data-centric work where retrieval is the centre of the problem.Skip when You need agent orchestration more than data access.
  • Pydantic AIframeworkMIT· AgentsType-safe agent framework that leans on Pydantic models.Use when Type-safe agents where schema correctness is the priority.Skip when You want a batteries-included ecosystem.
  • AutoGenframeworkMIT· AgentsMicrosoft research project for conversable multi-agent systems.Use when Research into conversable multi-agent systems.Skip when You need something stable and documented for production.
  • CrewAIframeworkMIT· AgentsRole-based orchestration where agents collaborate as a crew.Use when Role-based orchestration where agents are framed as a team.Skip when You want direct control over the loop itself.
  • Semantic KernelframeworkMIT· AgentsMicrosoft SDK for embedding AI steps into .NET and Python apps.Use when Embedding AI steps into an existing .NET estate.Skip when Your stack is not .NET.
  • MastraframeworkApache-2.0· AgentsTypeScript agent framework with typed workflows and evals.Use when TypeScript agents with typed workflows and evals built in.Skip when You are working in Python.also in evaluation-observability
  • OpenAI Agents SDKframeworkApache-2.0· AgentsLightweight primitives for handoffs, guardrails and tracing.Use when A small, legible set of primitives for handoffs and guardrails.Skip when You need multi-provider abstraction at every layer.also in guardrails-safety
  • AgnoframeworkMPL-2.0· AgentsMinimal agent runtime centered on model-agnostic tool interfaces.Use when A minimal agent runtime with model-agnostic tool interfaces.Skip when You need the wider ecosystem around it.
  • Claude Agent SDKframeworkproprietary· AgentsAnthropic's toolkit for building agents with tool use and hooks.Use when Building agents against Anthropic models, with tool use and hooks.Skip when You need provider independence at the framework layer.
  • LettaplatformApache-2.0· AgentsAgent runtime with persistent, editable memory as a first-class primitive.Use when Agents whose memory is a first-class, inspectable and editable state.Skip when You want memory to be a detail the framework handles for you.also in retrieval-vector-stores
  • smolagentsframeworkApache-2.0· AgentsMinimal code-first agent loop from the Hugging Face team.Use when Reading the whole agent loop in a page — it is deliberately tiny.Skip when You need a rich tool ecosystem around it.
  • Vercel AI SDKlibraryApache-2.0· AgentsProvider-agnostic building blocks for streaming apps and agents.Use when Streaming model calls in a TypeScript app, provider-agnostically.Skip when You need agent orchestration rather than model calls.
  • TemporalplatformMIT· WorkflowsDurable execution engine that survives crashes and long waits.Use when Work spanning minutes or hours that must survive restarts and retries.Skip when You need it live in a day — the adoption cost is real.
  • InngestplatformApache-2.0· WorkflowsEvent-driven step functions with durable replay for TypeScript.Use when Durable step functions in TypeScript, without running the engine.Skip when You are not working in TypeScript.
  • Trigger.devplatformApache-2.0· WorkflowsBackground jobs and AI workflows with long-running task support.Use when Long-running background jobs that must survive a deploy.Skip when You need more than TypeScript.
  • DagsterplatformApache-2.0· WorkflowsAsset-oriented orchestration for data and ML pipelines.Use when Data and ML pipelines framed as assets with explicit dependencies.Skip when You want a task queue rather than a lineage-aware orchestrator.
  • PrefectplatformApache-2.0· WorkflowsPython-native workflow orchestration with a managed cloud option.Use when Python-native flows with a managed cloud option behind them.Skip when You need deep lineage modelling.
  • Apache AirflowplatformApache-2.0· WorkflowsScheduler and DAG engine long used for batch data engineering.Use when Scheduled batch data engineering, where DAGs are the shared language.Skip when You are building a latency-sensitive product — it was not designed for one.
  • RestateplatformBSL-1.1· WorkflowsDurable execution with a low-latency stateful API surface.Use when Durable execution with a genuinely low-latency stateful API surface.Skip when You need the largest ecosystem around the engine.
  • DBOSlibraryApache-2.0· WorkflowsDurable workflows as ordinary Python functions and decorators.Use when Durable workflows as ordinary Python functions and decorators.Skip when You want a separate service to operate.
  • Fly Machinesplatformproprietary· WorkflowsHosting for stateful containers that fits durable agent workers.Use when Hosting stateful containers that fits durable agent workers.Skip when You want to own the infrastructure layer.
  • Modalplatformproprietary· WorkflowsServerless GPU and container platform popular for batch inference.Use when Serverless GPUs and containers for batch inference and scheduled jobs.Skip when Data locality rules forbid ephemeral compute.
  • NeMo GuardrailslibraryApache-2.0· GuardrailsProgrammable rails that constrain conversational flow.Use when Constraining conversational flow with programmable rails.Skip when You need deep semantic moderation — it is not a classifier.
  • Guardrails AIlibraryApache-2.0· GuardrailsValidators that check model output against a defined schema.Use when Validators that check output against a schema you define.Skip when You need free-form moderation rather than structural checks.
  • Llama GuardlibraryLlama-3.1-Community· GuardrailsSafety classifier for prompt and response moderation.Use when One classifier covering both prompt and response moderation.Skip when You need domain-specific policy enforcement.
  • Microsoft PresidiolibraryMIT· GuardrailsPII detection and anonymization for text and images.Use when Detecting and anonymising PII in text and images.Skip when Your data is already pseudonymised at source.
  • Lakera Guardserviceproprietary· GuardrailsPrompt injection and jailbreak detection at the gateway.Use when Prompt-injection and jailbreak detection at the gateway.Skip when Requests cannot leave your network.
  • garaklibraryApache-2.0· GuardrailsScanner that probes models for known vulnerability classes.Use when Probing your own system for known vulnerability classes.Skip when You have no remediation path for whatever it finds.
  • PyRITlibraryMIT· GuardrailsMicrosoft's red-team tool for generating and scoring attack prompts.Use when Generating and scoring attack prompts systematically.Skip when You are not authorised to test the target.
  • Invariant Guardrailsplatformproprietary· GuardrailsGuardrails as code, enforced inline in the call path.Use when Guardrails as code, enforced inline in the call path.Skip when Self-hosting is a hard requirement.
  • DSPylibraryMIT· PromptsDeclarative prompting that compiles to optimized programs from examples.Use when Compiling prompts into optimised programs from labelled examples.Skip when You have no examples and no way to score them.
  • InstructorlibraryMIT· PromptsSchema-constrained extraction with automatic validation and retry.Use when Schema-validated extraction with typed retry and partial streaming.Skip when You need a hard token-level guarantee rather than validation.
  • Humanloopplatformproprietary· PromptsPrompt versioning and evaluation for production teams.Use when Prompt versioning and evaluation run by a product team.Skip when Prompts or data cannot be sent to a vendor.
  • GuidancelibraryMIT· PromptsConstrained generation by interleaving control flow and model output.Use when Constrained decoding with control flow interleaved into generation.Skip when You need better ergonomics for plain extraction.
  • PromptLayerplatformproprietary· PromptsPrompt registry with request logging and regression testing.Use when A prompt registry with request logging and regression tests.Skip when You need self-hosting.
  • AgentaplatformApache-2.0· PromptsOpen-source prompt management with a playground and versioning.Use when Open-source prompt management with a playground and versioning.Skip when You need eval depth rather than prompt ergonomics.
  • Promptwatchplatformproprietary· PromptsPrompt regression tests and side-by-side comparison.Use when Prompt regression tests and side-by-side comparison.Skip when You need self-hosting.
  • TextGradlibraryMIT· PromptsAutomatic prompt optimisation via textual feedback loops.Use when Automatic prompt optimisation via textual feedback loops.Skip when Optimising without an eval set is guessing.
  • OutlineslibraryApache-2.0· PromptsConstrained generation that masks the token space to valid output.Use when Token-level constrained generation from a JSON schema or grammar.Skip when Post-hoc validation is enough for your use case.
  • LangSmithplatformproprietary· EvalsTracing, evaluation and dataset tooling across LangChain runs.Use when Teams already deep in LangChain needing tracing and datasets.Skip when Traces cannot leave your infrastructure.
  • Braintrustplatformproprietary· EvalsEvaluation platform with built-in scorers and a data flywheel.Use when Making evaluation, rather than tracing, the primary workflow.Skip when You need open formats and self-hosting.
  • Arize PhoenixlibraryApache-2.0· EvalsOpen-source tracing and evaluation built on OpenTelemetry.Use when OpenTelemetry-native evaluation you can run yourself.Skip when You want a vendor support contract behind it.
  • LangfuseplatformMIT· EvalsSelf-hostable LLM tracing, prompt management and cost analytics.Use when Self-hostable tracing, prompts and evals in one place.Skip when You want a fully managed product with a support contract.also in prompt-engineering
  • promptfoolibraryMIT· EvalsDeclarative red-teaming and regression tests for prompts and agents.Use when Declarative red-teaming and regression tests that run in CI.Skip when You need a managed UI rather than a test runner.
  • DeepEvallibraryApache-2.0· EvalsOpen-source pytest-style evaluation suite for LLM outputs.Use when pytest-style evaluation you can run next to your unit tests.Skip when You need a platform rather than a library.
  • Heliconeplatformproprietary· EvalsGateway-level observability with per-request cost and latency.Use when Gateway-level observability with per-request cost and latency.Skip when You need eval primitives more than traffic logs.also in routing-gateways
  • Weights & Biasesplatformproprietary· EvalsExperiment tracking and model registry with LLM eval surfaces.Use when Experiment tracking and a model registry, with eval surfaces on top.Skip when You need an open, exportable format.also in fine-tuning, workflow-orchestration
  • OpenTelemetrylibraryApache-2.0· EvalsVendor-neutral standard for emitting traces and metrics.Use when A vendor-neutral standard for emitting traces and metrics.Skip when You want a product with a user interface.
  • Vellumplatformproprietary· EvalsPrompt and eval platform with a visual debugger for chains.Use when Prompt and eval work with a visual debugger for chains.Skip when Self-hosting or reproducibility is a requirement.also in prompt-engineering
  • GentraceplatformApache-2.0· EvalsOpen-source tracing and dashboards for LLM applications.Use when Open-source tracing and dashboards for LLM applications.Skip when You want eval depth more than traffic visibility.
  • OpikplatformApache-2.0· EvalsOpen-source LLM observability and evaluation from Comet.Use when Open-source observability and evaluation you run yourself.Skip when You need enterprise support behind it.
  • Evidently AIlibraryApache-2.0· EvalsEvaluation and monitoring for both classical and LLM systems.Use when One evaluation model for both classical ML and LLM systems.Skip when You only need LLM-specific tracing.
  • Hugging Faceplatformproprietary· ReadingModel hub, datasets and open-source library ecosystem.Use when The model hub, the datasets and the open-source library ecosystem.Skip when You need a self-hosted artefact registry.
  • Full Stack Deep Learningreading· ReadingCourse covering the practical side of shipping LLM systems.Use when The practical side of shipping LLM systems, taught end to end.Skip when You want an API reference.
  • Sebastian Raschkareading· ReadingEssays and books distilling research into working knowledge.Use when Essays and books that distil research into working knowledge.Skip when You need primary sources.
  • arXivreading· ReadingPrimary preprint archive for machine learning research.Use when The primary preprint record for machine learning research.Skip when You need a curated and explained version.
  • Papers with Codereading· ReadingLinks papers to their reference implementations and benchmarks.Use when Linking papers to their reference implementations.Skip when You are offline — it is a live index.
  • Jay Alammarreading· ReadingVisual explanations of transformers, RAG and LLM internals.Use when Visual explanations of transformers, RAG and model internals.Skip when You want depth beyond the intuition.
  • AI Engineerreading· ReadingConference and community focused on shipping LLM products.Use when A community and conference focused on shipping LLM products.Skip when You need technical reference rather than practice.
  • Hugging Face Cookbookreading· ReadingTask-oriented notebooks for fine-tuning, serving and RAG.Use when Task-oriented notebooks for fine-tuning, serving and RAG.Skip when You need maintained library documentation.
  • Made With MLreading· ReadingCourse on designing, building and deploying ML systems.Use when Designing, building and deploying ML systems as a discipline.Skip when You only need LLM-specific material.
  • Distillreading· ReadingArchive of carefully explained model and method explainers.Use when Carefully explained model and method write-ups — though the publication is now archived.Skip when You need recent work; the archive stopped years ago.