180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers

Practical AI: Tools, Models & Frameworksinference

What changed

Hugging Face Transformers now supports expert streaming for Mixture-of-Experts (MoE) models, allowing a 180B-parameter model like Qwen3.8-Flash-Next to run on a 32GB Mac with disk-backed expert caching. The implementation uses a slot-pool cache strategy that enables the standard grouped_mm kernel to operate on streamed experts without custom kernels, achieving ~14 tokens/sec—within 6% of Strata's performance.

Why it matters

This makes large MoE models practically runnable on consumer hardware (single GPU or Mac) without specialized inference engines. For builders, it's now possible to run 180B models locally for coding tasks or local experimentation using standard Transformers APIs, with disk and quantization (3-bit experts) keeping memory footprint manageable.

How it compares

Unlike vLLM's expert offloading (which lacks caching and requires repeated PCIe transfers per token) or llama.cpp's approach (which thrashes the page cache when experts exceed RAM), this solution maintains a bounded device cache and prefetches based on routing patterns. It stays within 6% of Strata's specialized speed while remaining portable via standard Transformers, though both fall short of the deep PyTorch/hardware control needed to close the final gap.

Sources