Production & Governance
Whether it can be trusted, and whether it can be let go.
The question this role arrives with: “How do I know it still works, and what am I accountable for?”
25 tools · 6 sections · 3 also owned by another role
Routing & Gateways
all 10 →Fine-tuning & Training
all 12 →Guardrails & Safety
all 8 →- NeMo GuardrailslibraryApache-2.0
Use when Constraining conversational flow with programmable rails.
Skip when You need deep semantic moderation — it is not a classifier.
- Guardrails AIlibraryApache-2.0
Use when Validators that check output against a schema you define.
Skip when You need free-form moderation rather than structural checks.
- Llama GuardlibraryLlama-3.1-Community
Use when One classifier covering both prompt and response moderation.
Skip when You need domain-specific policy enforcement.
- Microsoft PresidiolibraryMIT
Use when Detecting and anonymising PII in text and images.
Skip when Your data is already pseudonymised at source.
- Lakera Guardserviceproprietary
Use when Prompt-injection and jailbreak detection at the gateway.
Skip when Requests cannot leave your network.
- garaklibraryApache-2.0
Use when Probing your own system for known vulnerability classes.
Skip when You have no remediation path for whatever it finds.
- PyRITlibraryMIT
Use when Generating and scoring attack prompts systematically.
Skip when You are not authorised to test the target.
- Invariant Guardrailsplatformproprietary
Use when Guardrails as code, enforced inline in the call path.
Skip when Self-hosting is a hard requirement.
Prompt Engineering
all 9 →Evaluation & Observability
all 13 →- LangSmithplatformproprietary
Use when Teams already deep in LangChain needing tracing and datasets.
Skip when Traces cannot leave your infrastructure.
- Braintrustplatformproprietary
Use when Making evaluation, rather than tracing, the primary workflow.
Skip when You need open formats and self-hosting.
- Arize PhoenixlibraryApache-2.0
Use when OpenTelemetry-native evaluation you can run yourself.
Skip when You want a vendor support contract behind it.
- LangfuseplatformMIT
Use when Self-hostable tracing, prompts and evals in one place.
Skip when You want a fully managed product with a support contract.
- promptfoolibraryMIT
Use when Declarative red-teaming and regression tests that run in CI.
Skip when You need a managed UI rather than a test runner.
- DeepEvallibraryApache-2.0
Use when pytest-style evaluation you can run next to your unit tests.
Skip when You need a platform rather than a library.
- Heliconeplatformproprietary
Use when Gateway-level observability with per-request cost and latency.
Skip when You need eval primitives more than traffic logs.
- Weights & Biasesplatformproprietary
Use when Experiment tracking and a model registry, with eval surfaces on top.
Skip when You need an open, exportable format.
- OpenTelemetrylibraryApache-2.0
Use when A vendor-neutral standard for emitting traces and metrics.
Skip when You want a product with a user interface.
- Vellumplatformproprietary
Use when Prompt and eval work with a visual debugger for chains.
Skip when Self-hosting or reproducibility is a requirement.
- GentraceplatformApache-2.0
Use when Open-source tracing and dashboards for LLM applications.
Skip when You want eval depth more than traffic visibility.
- OpikplatformApache-2.0
Use when Open-source observability and evaluation you run yourself.
Skip when You need enterprise support behind it.
- Evidently AIlibraryApache-2.0
Use when One evaluation model for both classical ML and LLM systems.
Skip when You only need LLM-specific tracing.
Learning & Reference
all 10 →Shared with another role
These are part of what you own and part of what someone else owns. That overlap is where the interesting arguments happen — who decides the eval threshold, who carries the pager.
Roles overlap on purpose, so these counts do not sum to the index. All 5 specialisations are listed in /roles.