FACTS Benchmark Suite: Systematically evaluating the factuality of large language models

What changed

Large language models (LLMs) are increasingly becoming a primary source for information delivery across diverse use cases, so it’s important that their responses are factually accurate. The FACTS Benchmark Suite Today, we’re teaming up with Kaggle to introduce the FACTS Benchmark Suite. Similar to our previous release, we are following standard industry practice and keeping an evaluation set held-out as a private set.

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• Introducing SimpleQA
• Introducing HELMET: Holistically Evaluating Long-context Language Models
• The Open Medical-LLM Leaderboard: Benchmarking Large Language Models in Healthcare

• Introducing SimpleQA similarity 0.22
• Introducing HELMET: Holistically Evaluating Long-context Language Models similarity 0.22
• The Open Medical-LLM Leaderboard: Benchmarking Large Language Models in Healthcare similarity 0.19