Quanto: a PyTorch quantization backend for Optimum

What changed

Reducing the number of bits means the resulting model requires less memory storage, which is crucial for deploying Large Language Models on consumer devices. It also enables specific optimizations for lower bitwidth datatypes, such as int8 or float8 matrix multiplications on CUDA devices. Many open-source libraries are available to quantize pytorch Deep Learning Models, each providing very powerful features, yet often restricted to specific model configurations and devices.

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• OpenAI standardizes on PyTorch
• Accelerated Inference with Optimum and Transformers Pipelines
• Introducing Optimum: The Optimization Toolkit for Transformers at Scale

• OpenAI standardizes on PyTorch similarity 0.29
• Accelerated Inference with Optimum and Transformers Pipelines similarity 0.20
• Introducing Optimum: The Optimization Toolkit for Transformers at Scale similarity 0.20