Finally, a Replacement for BERT: Introducing ModernBERT

What changed

This blog post introduces ModernBERT, a family of state-of-the-art encoder-only models representing improvements over older generation encoders across the board, with a 8192 sequence length, better downstream performance and much faster processing. ModernBERT is available as a slot-in replacement for any BERT-like models, with both a base (149M params) and large (395M params) model size. In this blog post we introduced the ModernBERT models, a new state-of-the-art family of small and efficient encoder-only models, finally giving BERT a much needed do-over.

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• Scaling-up BERT Inference on CPU (Part 1)
• Accelerate BERT inference with Hugging Face Transformers and AWS Inferentia
• Scaling up BERT-like model Inference on modern CPU - Part 2

• Scaling-up BERT Inference on CPU (Part 1) similarity 0.23
• Accelerate BERT inference with Hugging Face Transformers and AWS Inferentia similarity 0.19
• Scaling up BERT-like model Inference on modern CPU - Part 2 similarity 0.19