The efficient frontier of LLM inference

What changed

In the AI industry, we borrowed the term “efficient frontier” from economists. We use it to talk about managing tradeoffs, most often the tradeoff between cost and capabilities for models. However, quantization introduces a new set of tradeoffs between quality and serving efficiency.

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• Running a 28.9M parameter LLM on an $8 microcontroller
• AirLLM 70B inference with single 4GB GPU
• A $500 RL fine-tune of a 9B open model beat frontier models on catalog review

• Running a 28.9M parameter LLM on an $8 microcontroller similarity 0.48
• AirLLM 70B inference with single 4GB GPU similarity 0.46
• A $500 RL fine-tune of a 9B open model beat frontier models on catalog review similarity 0.45