DEV Community

Cover image for Hot experts, cold RAM: a big MoE on a box with more memory than VRAM
Private Local AI
Private Local AI

Posted on Originally published at localai.hex7.com

Hot experts, cold RAM: a big MoE on a box with more memory than VRAM

Two budgets, and only one of them is expensive

Every local deployment spends against two budgets at once: the size of the model and the price of
the memory holding it. VRAM is the one people reach for first, and it is the one that costs the most
per gigabyte. System DDR is fast, vast and cheap; GPU memory is fast, small and expensive. An
inference engine that can put weights in the second place changes the bill.

That is what Strata is for. It is forked from llama.cpp and built for memory efficiency: it keeps
resident only the experts that are hot - the ones actually in use - and leaves the rest parked.
That is how big MoE models run on small RAM.

Where llama.cpp ends and Strata starts

llama.cpp is the base of everything here: an inference engine in plain C/C++ that runs GGUF models on
hardware you already own - Metal, CUDA, Vulkan, SYCL or CPU only - with 1.5-8 bit quantisation and
hybrid offload for models too big for the GPU. Hybrid offload moves layers between CPU and GPU. An
MoE is a different shape of problem: it is not one network that is too big, it is a fleet of experts
of which only a few matter to a given query.

So the efficient move is not to place layers. It is to place experts, and to decide which of them
deserve to be resident right now. Strata does exactly that, and Avalon forks Strata to tune the same
hot set even tighter, which buys longer usable context and larger models on the same hardware.

DDR over VRAM

The second half of the picture is where the weights sit while they wait. The engine streams model
weights through ordinary system memory - fast, vast DDR instead of expensive GPU VRAM. One machine
with 256 GB of DDR and an RTX 5080 runs small and medium private AI comfortably.

For a small organisation the consequence is concrete: the machine you can afford is the machine that
fits the model you want. A 5080-class card with a lot of memory you already bought does more work
than a bigger card with less, and the upgrade path is RAM, not a new GPU.

Why the hot set holds

Expert selection is not random. Queries cluster: a support inbox about one product, a codebase, a
corpus of legal text. The gate keeps coming back to the same experts, so the hot set stabilises
after a while, page-in traffic drops away, and the parked experts stay parked - which is exactly the
efficiency argument for Mixture of Experts in the first place: each query activates only the experts
it needs, and the rest never load into memory at all.

What it asks of you

Two things. First, an NVMe that is actually fast: the cold tier is only cheap if the first read of an
expert is quick. Second, a model worth the complexity - a dense model that already fits in VRAM is
usually the simpler answer. And measure rather than trust: one model file, weights and prompt length
held identical across every engine that can load it on that box, reading and writing speed measured
apart, versions recorded with the number
(benchmark index).

The part that stays with you

Because inference runs on your own hardware inside the building, personal data never leaves the
premises. No Chapter V transfer to arrange, no standard contractual clauses to keep valid, no US
cloud disclosure to answer for - the accountability question never comes up, because the violation
never happens. Ask about an on-site assessment:
privatelocalai@proton.me.

Top comments (0)