Open LLM Leaderboard: DROP deep dive

What changed

DROP (Discrete Reasoning Over Paragraphs) is an evaluation where models must extract relevant information from English-text paragraphs before executing discrete reasoning steps on them (for example, sorting or counting items to arrive at the correct answer, see the table below for examples). The metrics used are custom f1 and exact match scores. We have therefore taken the decision to remove DROP from the Open LLM Leaderboard until a new version arises.

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• What's going on with the Open LLM Leaderboard?
• The Open Arabic LLM Leaderboard 2
• Introducing the Open Arabic LLM Leaderboard

• What's going on with the Open LLM Leaderboard? similarity 0.42
• The Open Arabic LLM Leaderboard 2 similarity 0.41
• Introducing the Open Arabic LLM Leaderboard similarity 0.40