OpenAI Evals

OpenAI Evals is a first-party evaluation ecosystem built by OpenAI, launched on March 14, 2023 alongside GPT-4. It covers three distinct surfaces: an MIT-licensed GitHub framework and community benchmark registry with 17,600+ stars, a programmatic Evals API for running evaluations against OpenAI model endpoints, and a no-code dashboard interface at platform.openai.com. The GitHub framework is the highest-starred open-source evaluation repository in DevExplore's Testing category research. Teams considering it for independent model evaluation should note that OpenAI both develops the models being evaluated and maintains the evaluation framework, and that data contributed to the public registry can be used by OpenAI for future model improvements.

Key Capabilities

• Community benchmark registry: A Git-LFS-backed registry of community-contributed evaluations covering factual accuracy, reasoning, instruction following, code generation, and domain-specific tasks, with OpenAI staff reviewing contributions that inform upcoming model development priorities

• Three eval templates: Basic string and fuzzy match checks, model-graded evaluations using an LLM judge configured via YAML rubrics, and custom Python-based evaluation logic for private use cases that require arbitrary scoring functions

• Evals API with dashboard visualization: Programmatic evaluation runs return granular per-criterion results with a report URL linking to visual breakdowns in the OpenAI platform, supporting both Chat Completions API and Responses API workflows

• No-code dashboard evals: Non-engineers configure and run evaluation suites directly in the OpenAI platform without writing code, using the same underlying infrastructure as the API and framework

• HealthBench domain benchmark: A health-domain evaluation suite developed with 262 physicians across 60 countries, released May 2026, available in the OpenAI GitHub repository for evaluating AI system performance on medical queries

• simple-evals benchmark runner: A companion MIT-licensed repository that runs OpenAI models against MMLU, HumanEval, GPQA, and MATH-500, used internally at OpenAI to track capability changes across model releases

See OpenAI Evals pricing details →