180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
What changed
Hugging Face Transformers now supports expert streaming for Mixture-of-Experts (MoE) models, allowing a 180B-parameter model like Qwen3.8-Flash-Next to run on a 32GB Mac with disk-backed expert caching. The implementation uses a slot-pool cache strategy that enables the standard grouped_mm kernel to operate on streamed experts without custom kernels, achieving ~14 tokens/sec—within 6% of Strata's performance.
Why it matters
This makes large MoE models practically runnable on consumer hardware (single GPU or Mac) without specialized inference engines. For builders, it's now possible to run 180B models locally for coding tasks or local experimentation using standard Transformers APIs, with disk and quantization (3-bit experts) keeping memory footprint manageable.
How it compares
Unlike vLLM's expert offloading (which lacks caching and requires repeated PCIe transfers per token) or llama.cpp's approach (which thrashes the page cache when experts exceed RAM), this solution maintains a bounded device cache and prefetches based on routing patterns. It stays within 6% of Strata's specialized speed while remaining portable via standard Transformers, though both fall short of the deep PyTorch/hardware control needed to close the final gap.
Sources
- 180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers (huggingface-blog)primary