Incredibly Fast BLOOM Inference with DeepSpeed and Accelerate
What changed
As the model needs 352GB in bf16 (bfloat16) weights (176*2), the most efficient set-up is 8x80GB A100 GPUs. Also 2x8x40GB A100s or 2x8x48GB A6000 can be used. The main reason for using these GPUs is that at the time of this writing they provide the largest GPU memory, but other GPUs can be used as well.
Why it matters
A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.
How it compares
Related prior coverage to compare against:
- Optimization story: Bloom inference
- Introducing 🤗 Accelerate
- Introducing The World's Largest Open Multilingual Language Model: BLOOM
Sources
- Incredibly Fast BLOOM Inference with DeepSpeed and Accelerate (huggingface-blog)primary