Fine-Tuning Mixture-of-Experts Models: Memory and Routing Realities

Overview

Mixture-of-experts architectures dominate the current generation of open-weight models, and they break the mental arithmetic most teams use for fine-tuning. A dense model’s memory requirement follows from its parameter count directly; an MoE model activates only a fraction of its parameters per token but must hold every expert resident, so memory follows total parameters while compute follows active parameters. A commonly cited example — an 8x7B MoE at FP16 — requires roughly 94 GB of VRAM for inference alone, which no single workstation card holds. This article works through the memory maths and the practical fine-tuning paths.

The memory arithmetic

Start with inference. An MoE model’s weights include every expert, whether or not a given token uses them, because routing is decided per token and any expert may be needed at any moment. So a model described as 8x7B holds roughly 47 billion parameters in total (experts share attention and other components), which at FP16 is around 94 GB — the figure commonly quoted for that architecture. Active parameters per token might be around 13 billion, which is what determines speed, not memory.

Fine-tuning multiplies this. Full fine-tuning adds gradients and optimiser state, typically bringing the total to roughly 12 to 16 bytes per trainable parameter. Applied to all experts, that puts a mid-size MoE well beyond a single node and into multi-node territory — which is why full fine-tuning of MoE models is rare outside well-resourced labs, and why the practical question is which subset to train.

Expert parallelism and its cost

The standard distributed strategy for MoE is expert parallelism: place different experts on different GPUs, and route tokens across the interconnect to whichever GPU holds the expert they need. This solves the memory problem elegantly — each GPU holds a fraction of the experts — but converts it into a communication problem, because every layer now requires an all-to-all exchange of tokens.

That all-to-all is the defining performance characteristic of MoE training. It is latency-sensitive and bandwidth-hungry, which makes interconnect quality far more important for MoE than for dense models of similar size. On a system with a wide NVLink domain the exchange stays inside the scale-up fabric; across a slower scale-out network it becomes the bottleneck. This is one of the clearest cases where the larger scale-up domain discussed in Vera Rubin NVL144 translates directly into throughput.

Routing imbalance is a real operational problem

The fourth row deserves emphasis for fine-tuning specifically. When you fine-tune an MoE model on a narrow domain, the router sends most tokens to a small subset of experts. Those experts adapt; the rest receive almost no gradient and drift, or simply remain unadapted, producing a model that behaves inconsistently when it encounters inputs that route elsewhere. Monitoring expert utilisation during fine-tuning is not optional.

Parameter-efficient paths

For most teams the answer is not to fine-tune the experts at all. LoRA-style low-rank adapters applied to attention and, selectively, to expert layers keep the base weights frozen, which removes optimiser state for the bulk of the model and makes the exercise fit on far less hardware. Because the base experts are unchanged, the routing-induced forgetting problem largely disappears.

Research has also explored combining the two ideas directly — treating LoRA adapters themselves as experts with token-level routing, which scales adapted capacity without duplicating the base model. These approaches remain less mature than plain LoRA on a dense model, and published work notes that multi-adapter schemes carry their own costs in parameter inflation and selection overhead. For a production team, plain adapter tuning on a frozen MoE base is the reliable starting point; the memory arithmetic for that approach is in LoRA and QLoRA memory math.

Practical sizing

Three cases. Adapter tuning on a quantized MoE base: the base at 4-bit brings a mid-size MoE within reach of a high-VRAM workstation or a single 8-GPU node, and adapter optimiser state is small. This covers most enterprise domain adaptation. Adapter tuning at higher precision on a full node: better quality, needs an 8-GPU server, still avoids full optimiser state.

Full fine-tuning across all experts: multi-node with expert parallelism, high-quality interconnect, and a real engineering effort to manage load balance. Reserve this for cases where adapters demonstrably fail, which for domain adaptation is uncommon. The general server sizing method is in fine-tuning GPU server sizing for enterprise LLMs.

What to measure before committing

Two measurements save most of the wasted effort. First, expert utilisation on your fine-tuning data: if a small subset of experts receives the overwhelming majority of tokens, a much smaller dense model may serve the domain equally well at a fraction of the deployment cost. Second, the actual quality gain from adaptation versus a well-constructed retrieval approach on the frozen base, since for knowledge-oriented tasks retrieval frequently outperforms fine-tuning at lower cost.

The deployment side deserves the same scrutiny. An MoE model’s memory footprint persists at serving time, so a model that is cheap to run per token can still be expensive to host, because every replica holds all experts. Factor that into the total cost rather than comparing active-parameter counts. The serving economics are developed in how much VRAM an LLM needs for inference.

Frequently asked questions