Opik by Comet

Opik is the open-source LLM evaluation platform built by Comet, a New York-based ML platform company founded in 2017 by Gideon Mendels and Nimrod Lahav. Comet spent seven years solving experiment tracking and reproducibility for traditional ML before launching Opik in September 2024, when LLM teams were reproducing the same undocumented chaos that MLOps had already fixed for classical models. That heritage shapes Opik's design: it covers tracing, evaluation, production monitoring, and agent observability in one platform rather than treating them as separate concerns. Comet has raised $63M across its funding rounds and operates with 103 employees, distinguishing it from single-product startups in the same Testing category.

Key Capabilities

• LLM and agent tracing: Deep instrumentation of LLM calls, RAG pipeline steps, tool invocations, and multi-agent workflows captures every decision point with full context, supporting frameworks including LangChain, LlamaIndex, AutoGen, Google ADK, and Flowise AI

• LLM-as-judge and automated evaluation: Scored evaluation runs measure relevance, factuality, coherence, and hallucination across prompt versions and model configurations, with experiment management tracking results across iterations

• Production monitoring with online evaluation rules: Live dashboards apply configurable evaluation rules to production traffic, alerting on quality regressions without waiting for offline test runs

• CI/CD integration with test case storage: Evaluation suites run on every prompt or configuration change, with stored test cases enabling reproducible regression comparisons across deploys

• Opik Agent Optimizer and Opik Guardrails: A production optimization layer that continuously improves agent behavior based on evaluation feedback, paired with guardrails that enforce quality and safety policies on live LLM outputs

• Self-hosted deployment and MCP integration: Docker and Kubernetes deployment options keep evaluation data within a team's own infrastructure, and the opik-mcp repository extends Opik into MCP-compatible AI agent workflows

See Opik by Comet pricing details →