What's going on with the Open LLM Leaderboard?

What changed

The discussion centered around one of the four evaluations displayed on the leaderboard: a benchmark for measuring Massive Multitask Language Understanding (shortname: MMLU). The community was surprised that MMLU evaluation numbers of the current top model on the leaderboard, the LLaMA model 🦙, were significantly lower than the numbers in the published LLaMa paper. We can use these probabilities to choose a token, for instance the most probable (or we can introduce some slight noise with a sampling to avoid having “too mechanical” answers).

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• The Open Arabic LLM Leaderboard 2
• Introducing the Open Arabic LLM Leaderboard
• Open LLM Leaderboard: DROP deep dive

• The Open Arabic LLM Leaderboard 2 similarity 0.59
• Introducing the Open Arabic LLM Leaderboard similarity 0.57
• Open LLM Leaderboard: DROP deep dive similarity 0.42