DEV Community

Cover image for The great sparsification: 2026 open-weight MoE models now activate 1 to 9 percent of their weights
ai maya
ai maya

Posted on

The great sparsification: 2026 open-weight MoE models now activate 1 to 9 percent of their weights

TL;DR

In 2026 the headline parameter count of an open-weight model stopped telling you how much compute it burns per token. The dominant design is a sparse Mixture of Experts (MoE) where the total weight budget keeps climbing into the trillions while the active fraction, the slice that actually runs for each token, keeps shrinking toward a few percent. DeepSeek's V4.1-Flash reportedly carries 552B total weights but activates roughly 8B during prefill. Kimi K3 carries 2.8T and activates about 104B. GLM-5.2 runs near 744B total with about 40B active. The practical consequence: compute scales with active parameters, but memory and loading cost scale with total parameters, and those two numbers now diverge by one to two orders of magnitude. If you deploy, finetune or budget for these models, the active/total split is the single number you should read first.

Why did open models split the parameter count in two?

A dense Transformer uses every weight for every token. A Mixture of Experts replaces the feed-forward block in each layer with many parallel "expert" subnetworks plus a router that, per token, picks a small handful to run. The attention layers stay dense and shared; only the expert banks are sparse.

That decoupling produces two very different numbers:

  • Total parameters decide how much memory you need to hold the model and how long it takes to load. Every expert has to be resident somewhere, even if it rarely fires.
  • Active parameters decide the floating-point work per token: active parameter count roughly tracks the FLOPs of a forward pass.

In a dense model these are the same number. In a 2026 MoE they are not even close. Here is the reported split across a representative set of recent open-weight releases:

Model Total params Active per token Active fraction Expert config (reported)
DeepSeek V4.1-Flash ~552B ~8B prefill / ~16B decode ~1.4% 1 shared + 384 routed, 6 routed active
Kimi K3 ~2.8T ~104B ~3.7% 16 of 896 experts
GLM-5.2 ~744B ~40B ~5.4% sparse MoE
MiniMax M3 ~428B ~23B ~5.4% sparse MoE
Qwen3.6-35B-A3B ~35B ~3B ~8.6% A3B MoE

Numbers are vendor and tracker reported, not independently re-measured. The trend is unambiguous: the bigger the total budget, the more aggressively the active fraction is pushed down.

What does a lower active fraction actually buy?

Think of it as paying for capacity once and compute per token separately.

A dense 104B model and Kimi K3 both do roughly 104B-parameters worth of arithmetic per token. But the dense model can only know 104B parameters worth of things, while Kimi K3 can store 2.8T parameters of specialized knowledge and route to the relevant 3.7% on demand. You get the knowledge capacity of a trillion-scale model at the per-token compute of a mid-size one.

A simple way to see the compute side:

def forward_flops_per_token(active_params_billion, n_tokens):
    # ~2 FLOPs per parameter per token for a forward pass
    return 2 * active_params_billion * 1e9 * n_tokens

# 1M-token context prefill
kimi   = forward_flops_per_token(104, 1_000_000)   # 2.08e17
flash  = forward_flops_per_token(8,   1_000_000)   # 1.6e16
print(f"Kimi K3 is {kimi/flash:.0f}x the prefill compute of V4.1-Flash")
# Kimi K3 is 13x the prefill compute of V4.1-Flash
Enter fullscreen mode Exit fullscreen mode

The flip side is memory. You still have to fit the whole expert bank:

def weight_memory_gb(total_params_billion, bytes_per_param=1):  # ~1 byte at int8/fp8
    return total_params_billion * 1e9 * bytes_per_param / 1e9

print(weight_memory_gb(2800))  # ~2800 GB at 8-bit for Kimi K3 total weights
print(weight_memory_gb(552))   # ~552 GB for V4.1-Flash
Enter fullscreen mode Exit fullscreen mode

So extreme sparsity turns a single question ("how big is it?") into two independent budgets. A model can be cheap to run and expensive to host at the same time.

How does sparsity change serving and the KV cache?

The active-parameter story is only half of serving cost. The other half is the KV cache, which grows with context length and is independent of expert sparsity. This is why 2026 MoE releases also attack the cache directly. DeepSeek V4.1-Flash reports a KV footprint around 890 bytes per token, which it claims is roughly 4x smaller than its predecessor, paired with a 1M-token context window.

Why that pairing matters: at 1M tokens, cache memory is often the binding constraint, not weights. A back-of-envelope comparison:

def kv_cache_gb(bytes_per_token, context_len):
    return bytes_per_token * context_len / 1e9

print(kv_cache_gb(890, 1_000_000))    # ~0.89 GB per sequence
print(kv_cache_gb(3600, 1_000_000))   # ~3.6 GB per sequence at 4x
Enter fullscreen mode Exit fullscreen mode

The difference decides how many concurrent long-context sessions a given accelerator can hold. Shrinking the active fraction cuts per-token FLOPs; shrinking bytes-per-token KV cache cuts how many users you can batch. Both levers are being pulled hard at once, and you have to read both to predict throughput.

Operationally, a few things follow from very sparse MoE:

  • Load time dominates cold starts. You still stream the full total weight budget off disk, so a 2.8T model is slow to spin up even though it is fast per token once warm.
  • Expert placement is a scheduling problem. With hundreds of experts across many devices, all-to-all routing traffic between GPUs can become the bottleneck, not matrix multiply.
  • Batch composition affects utilization. If tokens in a batch route to disjoint experts, you activate more distinct experts than the per-token count suggests, eroding the efficiency win.

Does a smaller active fraction hurt quality?

Not obviously, which is the surprising part. On reported public benchmarks the trillion-scale sparse models are competitive at the frontier: Kimi K3 is cited near the top of permissive open-weight GPQA rankings, and GLM-5.2 is reported to lead an open-weights intelligence index and to post strong software-engineering benchmark numbers. Qwen's A3B line shows the trend at the small end, with roughly 3B active parameters claimed to reach agentic-coding scores that dense models of similar active size do not.

The honest caveat: benchmark leadership rotates monthly, these figures come from vendors and third-party trackers, and active-parameter counts can be reported differently for prefill versus decode (V4.1-Flash is a clear example, roughly 8B versus 16B). Treat any single ranking as a snapshot, and when you compare two MoE models, confirm whether the "active" number refers to the same phase.

How should I read a model card in 2026?

A short checklist that falls out of all this:

  1. Find both numbers. Total parameters and active parameters per token. If a card only gives one, you cannot estimate either cost.
  2. Separate your two budgets. Compute and latency track active params; memory, hosting and cold-start track total params.
  3. Check the KV cache spec. Bytes per token times context length often dominates memory at long context, independent of expert sparsity.
  4. Note prefill versus decode. Some models activate different expert counts in each phase; use the right one for your workload (long documents are prefill-heavy, chat is decode-heavy).
  5. Mind the expert topology. Shared experts, routed experts and how many fire per token determine routing traffic and real utilization under batching.

FAQ

Is a model with fewer active parameters always cheaper to run?
Cheaper in per-token compute, yes. But not cheaper to host: you still need memory for all total parameters, and a very large total budget means slow loading and higher fixed infrastructure cost. The two budgets move independently.

Why keep weights you rarely use?
Because capacity and compute are being deliberately separated. A large expert bank stores specialized knowledge; the router pays only for the slice each token needs. You buy trillion-scale knowledge at mid-size per-token cost, at the price of trillion-scale memory.

What is the difference between prefill and decode active parameters?
Prefill processes the whole prompt in parallel; decode generates one token at a time. Some 2026 MoE models activate different expert counts in each phase, so a single "active parameters" figure can be ambiguous. Match the number to whether your workload is prompt-heavy or generation-heavy.

Does extreme sparsity reduce output quality?
Reported benchmarks suggest the top sparse models remain at the frontier, but rankings shift frequently and most numbers are vendor or tracker reported rather than independently reproduced. Verify on your own task before committing.

How do I estimate serving capacity for a long-context MoE?
Add two terms: resident weight memory (total params times bytes per param) plus KV cache (bytes per token times context length times concurrent sequences). At 1M-token context the KV term frequently dominates, so the per-token cache spec can matter more than the active fraction.

Is the sparsification trend likely to continue?
The direction is clear in 2026: total budgets climbing while active fractions fall toward a few percent, alongside aggressive KV-cache compression. Whether it holds depends on routing quality and interconnect bandwidth, both of which become the real bottlenecks as expert counts grow.

Further reading

Top comments (0)