Inspect AI
Evaluate frontier AI models for dangerous capabilities in sandboxed environments
Inspect AI is profiled here as a Evaluation tool for engineering teams. Read about features, pricing, and how it compares to related options in the tools directory.
Description
Inspect is an open-source evaluation framework developed by the UK AI Security Institute (AISI) and Meridian Labs, first open-sourced in May 2024 following the establishment of AISI at the Bletchley Park AI Safety Summit in November 2023. Unlike every other tool in the Testing category, Inspect was built to serve a government mandate: giving independent evaluators the infrastructure to assess frontier models for dangerous capabilities without relying on self-reported safety claims from the model developers themselves. The framework is MIT-licensed, runs across all major frontier model providers through a single interface, and is the mandatory evaluation framework for all UK AISI Autonomous Systems assessments.
Key Capabilities
Sandboxed agent evaluation: Untrusted code and agent behaviors run in Docker, Kubernetes, or Proxmox sandboxes with domain and network controls, tool approval gating, and isolated scaffolding servers, designed specifically for testing potentially dangerous agent capabilities safely
External agent support: Inspect evaluates autonomous coding agents including Claude Code, Codex CLI, and Gemini CLI as external agents, along with multi-agent compositions built on AutoGen, LangChain, or custom scaffolds
200+ pre-built evaluations: A community-maintained registry covering agentic AI security vulnerabilities, mathematics benchmarks including AIME 2024 through 2026, autonomous harmful behavior assessments, and capability evaluations contributed by AI safety institutes and frontier labs
Broad provider coverage: A single task interface runs against OpenAI, Anthropic, Google, Mistral, xAI, AWS Bedrock, Azure AI, Together, Cloudflare, and local models via vLLM, Ollama, and llama-cpp without changing evaluation logic
Inspect View and VS Code extension: A web-based log viewer monitors and visualizes evaluation runs, and a VS Code extension supports authoring and debugging evaluation tasks without leaving the development environment
Python-extensible task architecture: Evaluations compose datasets, solvers, and scorers as Python objects, with MCP tool support, built-in bash and web browsing tools, and an extension API for new elicitation and scoring techniques
Alternative tools
- Gentrace
Testing and evaluation for generative AI applications
- HELM
Reproducible, multi-scenario benchmarking of foundation models
- lm-evaluation-harness
Standard framework for benchmarking language models
- garak
Vulnerability scanner for large language models
- DeepChecks
Validate ML models, LLM applications, and AI agent decisions across every development stage
- Evidently AI
Evaluate, test, and monitor traditional ML models and LLM applications from one framework