Inspect AI
Evaluate frontier AI models for dangerous capabilities in sandboxed environments
Description
Inspect is an open-source evaluation framework developed by the UK AI Security Institute (AISI) and Meridian Labs, first open-sourced in May 2024 following the establishment of AISI at the Bletchley Park AI Safety Summit in November 2023. Unlike every other tool in the Testing category, Inspect was built to serve a government mandate: giving independent evaluators the infrastructure to assess frontier models for dangerous capabilities without relying on self-reported safety claims from the model developers themselves. The framework is MIT-licensed, runs across all major frontier model providers through a single interface, and is the mandatory evaluation framework for all UK AISI Autonomous Systems assessments.
Key Capabilities
Sandboxed agent evaluation: Untrusted code and agent behaviors run in Docker, Kubernetes, or Proxmox sandboxes with domain and network controls, tool approval gating, and isolated scaffolding servers, designed specifically for testing potentially dangerous agent capabilities safely
External agent support: Inspect evaluates autonomous coding agents including Claude Code, Codex CLI, and Gemini CLI as external agents, along with multi-agent compositions built on AutoGen, LangChain, or custom scaffolds
200+ pre-built evaluations: A community-maintained registry covering agentic AI security vulnerabilities, mathematics benchmarks including AIME 2024 through 2026, autonomous harmful behavior assessments, and capability evaluations contributed by AI safety institutes and frontier labs
Broad provider coverage: A single task interface runs against OpenAI, Anthropic, Google, Mistral, xAI, AWS Bedrock, Azure AI, Together, Cloudflare, and local models via vLLM, Ollama, and llama-cpp without changing evaluation logic
Inspect View and VS Code extension: A web-based log viewer monitors and visualizes evaluation runs, and a VS Code extension supports authoring and debugging evaluation tasks without leaving the development environment
Python-extensible task architecture: Evaluations compose datasets, solvers, and scorers as Python objects, with MCP tool support, built-in bash and web browsing tools, and an extension API for new elicitation and scoring techniques
Alternative tools
- OpenAI Playground
Browser-based prompt iteration environment for the OpenAI API.
- Confident AI
The cloud platform built on DeepEval, the pytest-compatible LLM testing framework
- Galileo AI
Detect hallucinations and agent failures across the full development lifecycle
- LangWatch
Open-source LLMOps platform for observability, evaluation, and agent simulation.
- Adaline
End-to-end prompt management platform covering iteration, evaluation, deployment, and monitoring.
- Maxim AI
End-to-end AI evaluation platform with pre-production agent simulation and production observability