Introducing HELMET: Holistically Evaluating Long-context Language Models

What changed

Introducing HELMET: Holistically Evaluating Long-context Language Models Paper: https://arxiv.org/abs/2410.02694 Website: https://princeton-nlp.github.io/HELMET Code & Data: https://github.com/princeton-nlp/HELMET Since we first released HELMET last October, there has been more development on long-context language models than ever before, and we are thrilled to see the adoption of HELMET by the community, such as Microsoft's Phi-4 and AI21's Jamba 1.6. After the initial release, we have added more models to our evaluation suite and conducted additional analyses. We are excited to share our new results and present HELMET at ICLR 2025!

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• FACTS Benchmark Suite: Systematically evaluating the factuality of large language models
• Welcome Gemma 3: Google's all new multimodal, multilingual, long context open LLM
• Introducing NVIDIA Nemotron 3 Nano Omni: Long-Context Multimodal Intelligence for Documents, Audio and Video Agents

• FACTS Benchmark Suite: Systematically evaluating the factuality of large language models similarity 0.22
• Welcome Gemma 3: Google's all new multimodal, multilingual, long context open LLM similarity 0.21
• Introducing NVIDIA Nemotron 3 Nano Omni: Long-Context Multimodal Intelligence for Documents, Audio and Video Agents similarity 0.18