Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA

What changed

As we strive to make models even more accessible to anyone, we decided to collaborate with bitsandbytes again to allow users to run models in 4-bit precision. This is a new method introduced today in the QLoRA paper by Dettmers et al. Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU.

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• Introducing AutoRound: Intel’s Advanced Quantization for LLMs and VLMs
• Fine-tuning LLMs to 1.58bit: extreme quantization made easy
• Bringing Nunchaku 4-bit Diffusion Inference to Diffusers

• Introducing AutoRound: Intel’s Advanced Quantization for LLMs and VLMs similarity 0.20
• Fine-tuning LLMs to 1.58bit: extreme quantization made easy similarity 0.19
• Bringing Nunchaku 4-bit Diffusion Inference to Diffusers similarity 0.17