Open-source LLM evaluation and observability
Trace, evaluate and monitor LLM apps with LLM-as-judge metrics you can self-host.
Opik is the open-source LLM evaluation platform from Comet: tracing of every LLM call with inputs, outputs and costs, datasets and experiment tracking, LLM-as-judge evaluators, hallucination and moderation checks, and a comparison view for prompt and model iterations.
Why founders use it: the failure mode of AI features is silent quality decay, a prompt tweak that fixed one case and broke three, and Opik turns that into a measurable pipeline, every change scored against a dataset before it ships. The tracing works across frameworks with a few lines of integration, and self-hosting keeps the transcripts, your customers' conversations, on infrastructure you control.
A realistic setup: the SDK around model calls, traces per feature and user, a golden dataset of real cases, judge evaluators run on every candidate prompt, and a rule that nothing reaches production without beating the incumbent's scores.
Trade-offs: judge models need calibration against human labels, the self-hosted stack wants containers you babysit, and the space moves fast enough that workflows migrate. Compare with Langfuse for observability breadth, Braintrust for eval-first polish, and promptfoo for the lightweight CLI take.
Who it's for
Agent developers who need tracing plus evaluation in one loop and want the option to self-host.
Official links
Keep exploring
Concepts in the glossary
2 in catalog
Opik is an alternative to
5 in catalog