FairStress: A deeper evaluation of fairness in LLM decision-making

What changed

📄 Tech Report | 🤗 Data | 💻 Code Today we’re introducing FairStress, a benchmark for measuring demographic fairness in LLMs beyond the choice a model ultimately makes in decisions impacting people. FairStress is available for download, and we’re also building it into Olmo-eval, our open workbench for running and comparing model evaluations. Standard demographic fairness benchmarks often test whether a model selects the same candidate in otherwise identical scenarios after the candidates’ demographic identities are swapped.

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• Introducing the Open Ko-LLM Leaderboard: Leading the Korean LLM Evaluation Ecosystem
• Introducing Decision Transformers on Hugging Face 🤗
• Rethinking LLM Evaluation with 3C3H: AraGen Benchmark and Leaderboard

• Introducing the Open Ko-LLM Leaderboard: Leading the Korean LLM Evaluation Ecosystem similarity 0.19
• Introducing Decision Transformers on Hugging Face 🤗 similarity 0.16
• Rethinking LLM Evaluation with 3C3H: AraGen Benchmark and Leaderboard similarity 0.15