Introducing the Open Chain of Thought Leaderboard

What changed

Unlike most performance based leaderboards, we’re not scoring the absolute accuracy a model achieves on a given task, but the difference between the accuracy with and without chain-of-thought prompting: accuracy gain Δ = accuracy with CoT – accuracy w/o CoT. This allows us to truly inspect the impact that chain-of-thought has on model accuracy. Note: without CoT prompting, we use the loglikelihood accuracy to score the model on multiple choice evaluation.

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• Introducing the Open Arabic LLM Leaderboard
• Introducing the Open FinLLM Leaderboard
• What's going on with the Open LLM Leaderboard?

• Introducing the Open Arabic LLM Leaderboard similarity 0.40
• Introducing the Open FinLLM Leaderboard similarity 0.39
• What's going on with the Open LLM Leaderboard? similarity 0.36