OpenAI’s Jalapeño chip is built for fast inference at scale, benchmarks show

What changed

At the Hot Chips conference on Tuesday, OpenAI shared a more detailed look at Jalapeño, including the first batch of benchmark results for the new system. Tested on SemiAnalysis’ InferenceX benchmark, Jalapeño registered both more tokens per user and more throughput per kilowatt than the currently available state-of-the-art inference processors. “The bottom line is that the results show a very, very significant performance advance over state of the art,” said Richard Ho, OpenAI’s head of hardware, in a press call.

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• Jalapeño’s first results show industry-leading speed and efficiency in AI inference
• OpenAI and Broadcom unveil LLM-optimized inference chip
• StarCoder: A State-of-the-Art LLM for Code

• Jalapeño’s first results show industry-leading speed and efficiency in AI inference similarity 0.35
• OpenAI and Broadcom unveil LLM-optimized inference chip similarity 0.25
• StarCoder: A State-of-the-Art LLM for Code similarity 0.14