Fine-tuning Llama 2 70B using PyTorch FSDP

What changed

In this blog post, we will look at how to fine-tune Llama 2 70B using PyTorch FSDP and related best practices. We will be leveraging Hugging Face Transformers, Accelerate and TRL. We will also learn how to use Accelerate with SLURM.

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• Accelerating PyTorch distributed fine-tuning with Intel technologies
• OpenAI standardizes on PyTorch
• Fine-tune Llama 2 with DPO

• Accelerating PyTorch distributed fine-tuning with Intel technologies similarity 0.32
• OpenAI standardizes on PyTorch similarity 0.28
• Fine-tune Llama 2 with DPO similarity 0.27