promptfoo

Reading on evals

• Fix the ranking, not the promptWhen a RAG system gives a bad answer, the instinct is to rewrite the prompt. Nine times out of ten the prompt is fine and the context was never there.
• Evals are the asset, not the loggingTraces tell you what happened. Evals tell you whether it got better. Most teams build the first and mistake it for the second.
• Build a cost model before you optimiseTeams optimise the thing they can see in the dashboard. The dashboard shows tokens. The number that matters is cost per successful outcome, and nobody is measuring it.
• Observability is not loggingA log line with a status code is not a trace. The difference is whether it can reconstruct a decision after the model that made it has been replaced twice.

• DeepEvalOpen-source pytest-style evaluation suite for LLM outputs.
• garakScanner that probes models for known vulnerability classes.

Listed as an alternative to

• DeepEvalDeepEvalOpen-source pytest-style evaluation suite for LLM outputs.Use when pytest-style evaluation you can run next to your unit tests.Skip when You need a platform rather than a library.Apache-2.0 · self-hosted · free
• Evidently AIEvidently AIEvaluation and monitoring for both classical and LLM systems.Use when One evaluation model for both classical ML and LLM systems.Skip when You only need LLM-specific tracing.Apache-2.0 · self-hosted · free

Also in evals

• LangSmithLangSmithTracing, evaluation and dataset tooling across LangChain runs.Use when Teams already deep in LangChain needing tracing and datasets.Skip when Traces cannot leave your infrastructure.proprietary · saas · usage-based
• BraintrustBraintrustEvaluation platform with built-in scorers and a data flywheel.Use when Making evaluation, rather than tracing, the primary workflow.Skip when You need open formats and self-hosting.proprietary · saas · usage-based
• Arize PhoenixArize PhoenixOpen-source tracing and evaluation built on OpenTelemetry.Use when OpenTelemetry-native evaluation you can run yourself.Skip when You want a vendor support contract behind it.Apache-2.0 · self-hosted · free
• LangfuseLangfuseSelf-hostable LLM tracing, prompt management and cost analytics.Use when Self-hostable tracing, prompts and evals in one place.Skip when You want a fully managed product with a support contract.MIT · self-hosted · free-tier
• DeepEvalDeepEvalOpen-source pytest-style evaluation suite for LLM outputs.Use when pytest-style evaluation you can run next to your unit tests.Skip when You need a platform rather than a library.Apache-2.0 · self-hosted · free
• HeliconeHeliconeGateway-level observability with per-request cost and latency.Use when Gateway-level observability with per-request cost and latency.Skip when You need eval primitives more than traffic logs.proprietary · saas · free-tier
• Weights & BiasesWeights & BiasesExperiment tracking and model registry with LLM eval surfaces.Use when Experiment tracking and a model registry, with eval surfaces on top.Skip when You need an open, exportable format.proprietary · saas · usage-based
• OpenTelemetryOpenTelemetryVendor-neutral standard for emitting traces and metrics.Use when A vendor-neutral standard for emitting traces and metrics.Skip when You want a product with a user interface.Apache-2.0 · self-hosted · free
• VellumVellumPrompt and eval platform with a visual debugger for chains.Use when Prompt and eval work with a visual debugger for chains.Skip when Self-hosting or reproducibility is a requirement.proprietary · saas · usage-based
• GentraceGentraceOpen-source tracing and dashboards for LLM applications.Use when Open-source tracing and dashboards for LLM applications.Skip when You want eval depth more than traffic visibility.Apache-2.0 · self-hosted · free-tier
• OpikOpikOpen-source LLM observability and evaluation from Comet.Use when Open-source observability and evaluation you run yourself.Skip when You need enterprise support behind it.Apache-2.0 · self-hosted · free
• Evidently AIEvidently AIEvaluation and monitoring for both classical and LLM systems.Use when One evaluation model for both classical ML and LLM systems.Skip when You only need LLM-specific tracing.Apache-2.0 · self-hosted · free