Fine-tune video and image models at scale with NVIDIA NeMo Automodel and 🤗 Diffusers
What changed
NVIDIA and Hugging Face released an integration that lets NeMo Automodel—a PyTorch DTensor-native training library—fine-tune any Diffusers-format model directly from the Hub with no checkpoint conversion. It supports flow-matching diffusion models including FLUX.1-dev (12B), Wan 2.1, and HunyuanVideo (13B), with latent-space training via pre-encoded VAE outputs and multiresolution bucketed dataloading. Both full fine-tuning and LoRA/PEFT are supported through YAML-driven recipes, and parallelism (FSDP2, tensor, context, pipeline, expert) is a config choice rather than a code rewrite. It's Apache 2.0, ships as a Docker container or pip install, and supports SLURM multi-node orchestration today.
Why it matters
Builders can scale diffusion fine-tuning from one GPU to multi-node clusters without rewriting models or juggling separate training formats, and checkpoints load straight back into DiffusionPipeline for inference. This makes training large image and video models like FLUX.1-dev and HunyuanVideo practical while keeping downstream tooling (quantization, compilation, LoRA, custom samplers) intact.
How it compares
This extends the earlier NeMo Automodel transformer fine-tuning work into the diffusion/Diffusers ecosystem, applying the same Hub-native, config-driven, no-conversion approach to image and video generation models rather than language transformers.
Sources
- Fine-tune video and image models at scale with NVIDIA NeMo Automodel and 🤗 Diffusers (huggingface-blog)primary