Fixing Open LLM Leaderboard with Math-Verify

What changed

Open LLM Leaderboard Track, rank and evaluate open LLMs and chatbots Today, we’re thrilled to share that we’ve used Math-Verify to thoroughly re-evaluate all 3,751 models ever submitted to the Open LLM Leaderboard, for even fairer and more robust model comparisons! The Open LLM Leaderboard is the most used leaderboard on the Hugging Face Hub: it compares open Large Language Models (LLM) performance across various tasks. One of these tasks, called MATH-Hard, is specifically about math problems: it evaluates how well LLMs solve high-school and university-level math problems.

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• What's going on with the Open LLM Leaderboard?
• The Open Arabic LLM Leaderboard 2
• Introducing the Open Arabic LLM Leaderboard

• What's going on with the Open LLM Leaderboard? similarity 0.39
• The Open Arabic LLM Leaderboard 2 similarity 0.38
• Introducing the Open Arabic LLM Leaderboard similarity 0.37