Score quality with repeatable tests
The test suites for AI systems: datasets, judges and scoring loops that catch quality regressions before users do.
This hub gathers the blocks for evaluating AI systems: eval frameworks, dataset tooling, LLM-as-judge services and the observability platforms whose trace-and-score loops turn model behavior from vibes into measurements.
Why it matters to an indie builder: every prompt, model or retrieval change is a release with no compiler to catch regressions. Evals are the test suite for that world, a fixed set of real inputs with expected outcomes, run on every change, catching the silent quality decay that user complaints discover weeks later. Teams that skip evals refactor prompts blindly; teams with them ship changes hourly.
A realistic loop: collect representative cases from production traces, define success as assertions or a rubric score, run the suite on each candidate prompt or model, and promote only what beats the incumbent on quality and cost. Over time the dataset becomes the product's real specification.
Trade-offs: writing good cases is labor, judge models have their own biases, and scores drift from reality without periodic human review. Compare the frameworks here by how they handle non-determinism, and see the observability blocks for where the raw cases come from.
Who it's for
AI teams shipping changes who need signal on quality beyond vibes.
Keep exploring
10 in catalog
Evals is featured by
-
Opik
Open-source LLM evaluation and observability
LLM Ops -
Weights & Biases
The AI developer platform
LLM Ops -
Langfuse
The open-source LLM engineering platform
LLM Ops -
LangSmith
Build trustworthy AI agents
LLM Ops -
Braintrust
Enterprise AI evaluation
LLM Ops -
Arize Phoenix
Open-source AI observability
LLM Ops -
DeepEval
The open-source LLM evaluation framework
LLM Ops -
promptfoo
Test your LLM app before you ship
LLM Ops -
W&B Weave
Observability and continuous improvement for production agents
LLM Ops -
Agenta
The open-source workspace for your agents
LLM Ops
1 in catalog