HELM

 HELM, Holistic Evaluation of Language Models, is an open-source benchmarking project from Stanford's Center for Research on Foundation Models. It evaluates models across many scenarios and reports calibration, resilience, fairness, and efficiency alongside accuracy, so a comparison reflects several dimensions of behavior at once. The project publishes living leaderboards with full transparency into prompts, predictions, and results. Its emphasis on transparency lets anyone inspect the exact prompts and predictions behind a score, which supports independent verification. Specialized leaderboards extend the methodology to medicine, safety, and vision-language models.

Key Capabilities:

• Multi-metric evaluation spanning accuracy, resilience, and fairness

• Broad scenario coverage across tasks and domains

• Specialized leaderboards for medicine, safety, and vision-language models

• Transparent records of prompts, raw predictions, and scores

• Standardized methodology for reproducible comparison

• Apache 2.0 framework with publicly hosted results