Introduction: What Is “Humanity’s Last Exam”?

“Humanity’s Last Exam” (HLE) is the most ambitious AI benchmark ever created, comprising 2,500 questions across mathematics, humanities, natural sciences, and more. It combines multiple-choice and exact-match short-answer formats, with nearly a quarter of all questions requiring both text and image analysis. Designed by the Center for AI Safety in partnership with Scale AI, HLE tests whether contemporary AI systems can truly match or exceed expert human reasoning.

Origins and Development

The idea originated from a discussion between Dan Hendrycks (Center for AI Safety) and Elon Musk, who argued that existing benchmarks were no longer challenging enough for state-of-the-art models. To address this, they launched a global crowdsourcing campaign: experts submitted their toughest questions, which underwent rigorous peer review. Top contributors earned cash rewards from a half?million?dollar prize pool, ensuring high?quality, novel content.

Composition and Coverage

HLE’s questions are distributed across key disciplines: mathematics (41%), computer science & AI (10%), biology & medicine (11%), physics (9%), humanities & social sciences (9%), chemistry (7%), engineering (4%), and other topics (9%). Formats include 24% multimodal items, 24% multiple-choice, and 76% exact-match short answers, offering a comprehensive measure of both knowledge recall and complex reasoning.

Significance: Benchmark Saturation and AI Safety

As AI models rapidly saturate existing tests—achieving near?perfect scores within months—HLE offers a fresh, robust challenge that resists overfitting. By keeping half the questions private for final evaluation, organizers ensure the benchmark remains a true test of general intelligence rather than memorized training data. This integrity is vital for tracking real progress and guiding safe, responsible AI development.

Initial Results and Implications

Early leaderboards reveal top models scoring around 20% accuracy—far below typical human expert performance—highlighting HLE’s difficulty. This gap underscores the need for benchmarks that push beyond superficial capabilities and drive innovation toward genuine reasoning and understanding in AI systems.

Future Directions

Organizers plan to update HLE regularly with fresh, harder questions and expand to include audio and video items, maintaining its status as the definitive “last exam.” Success here could inspire similar domain?specific benchmarks in fields like medicine, law, and engineering, further anchoring AI advances in rigorous, expert?level evaluation.