One thing I'd be curious to see in the next round is how context depth changes the offloading results. A setup that looks fine at 512 tokens might behave very differently once the KV cache grows and starts competing for VRAM with the model weights.
I'd run a few llama-bench tests with fixed generation lengths but different context depths, using -d and -pg to separate prompt processing from generation. That could help distinguish whether the slowdown is coming from expert weight transfers, attention/KV-cache pressure, or the interaction between them.
Especially with LFM, where the 100-token and 512-token results told quite different stories, I think that would be a useful next experiment.
Developer, maker, and perpetual project starter. I build web apps and dev tools mostly, things that I think are really useful and should exist and be simple to use. I also like to experiment with AI.
I actually did some context scaling tests with Phi-3.5 and Apex in my original experiments, going from 4K to 16K, but those were changes to the configured context window rather than tests with prompts that actually filled it.
I haven't explored how growing the actual context affects the different offloading strategies yet, and I think that would be really interesting, especially with the differences I saw between the shorter and longer LFM runs.
Using llama-bench to separate prompt processing and generation at different context depths sounds like a good next step. I'd love to see whether the KV cache starts affecting which offloading configuration performs better as the context grows.
Thanks for the suggestion! My list of things to test keeps getting longer.
That distinction between configured context size and actually filling the context is important. A 16K context window doesn't necessarily tell us much about memory pressure if the prompt only uses a fraction of it.
I'd be particularly curious about what happens when the KV cache starts competing with expert weights for GPU memory. In an MoE setup, that could change the offloading trade-off quite noticeably, especially if some experts are already being moved between CPU and GPU during generation.
It'd be interesting to see whether the crossover point between different offloading strategies shifts as the actual context grows. That might explain some of the differences you observed in the LFM runs.
Seeing someone revisit their own conclusions publicly is interesting to me, especially when the newer results complicate the original explanation. That part of research doesn't get discussed enough.
Developer, maker, and perpetual project starter. I build web apps and dev tools mostly, things that I think are really useful and should exist and be simple to use. I also like to experiment with AI.
I think it's easy to get attached to an explanation once you've written it all out, but sometimes the more you investigate, the more complicated things get. I actually enjoyed discovering that some of my original assumptions weren't quite right. It gave me a lot more to explore, and I learned more from going back and questioning the results than I probably would have otherwise.
Your reply reminds me of that old Bertrand Russell observation about intelligent people being full of doubt. Iâve always found that interesting because the deeper you investigate something, the more you realize how much you might be missing. Being willing to question your own conclusions seems like one of the most important parts of research.
The early-ending Apex run makes me want actual output-token counts beside each timing. Were the 512-token tests all reaching 512, or was that just the limit? That would help tell whether the longer runs are measuring comparable amounts of work before digging into why they slowed down.
Developer, maker, and perpetual project starter. I build web apps and dev tools mostly, things that I think are really useful and should exist and be simple to use. I also like to experiment with AI.
That's a really good point, and something I should have documented more carefully. The 512 tokens was the configured generation limit, but I didn't include the actual output-token counts for each run in my results.
The early-ending Apex run made me realize how much that can affect a short benchmark, and you're right that it's important to verify the longer runs were actually doing comparable amounts of work too.
I think my next step is going to be logging the actual generated token counts alongside the timing data, rather than relying on the configured limit. Thanks for pointing that out.
The 'a measurement and an explanation aren't the same thing' is the most important line in this article. So many benchmark write-ups stop at the number and never dig into why it happened. The fact that you went back and re-tested after feedback makes this much more credible.
Developer, maker, and perpetual project starter. I build web apps and dev tools mostly, things that I think are really useful and should exist and be simple to use. I also like to experiment with AI.
Thank you! That's actually one of the biggest things I've taken away from all this. It's one thing to observe a performance difference, but understanding exactly what's causing it is a whole different challenge. And I've definitely caught myself coming up with explanations that sounded perfectly reasonable until I did more testing. I'm glad I went back and revisited the results. It's been a really interesting learning experience.
This level of intellectual honesty is rare to see in benchmark posts. It takes a lot of discipline to go back, rerun the tests, and publicly admit that the initial "perfectly reasonable" narrative was wrong. It's a great reminder that performance metrics are just numbers until you actually understand the mechanism behind them.
I'm curious â when you were digging into the data this time around, did you find that the MoE offloading behavior was more dependent on the specific prompt structure, or was it mostly the hardware memory bandwidth limits doing the heavy lifting?
Developer, maker, and perpetual project starter. I build web apps and dev tools mostly, things that I think are really useful and should exist and be simple to use. I also like to experiment with AI.
Thank you. I really appreciate that. I think one of the most interesting parts of experimenting with things like this is discovering that an explanation you were pretty confident about doesn't necessarily hold up once you start testing further.
As for the prompt structure versus memory bandwidth, I haven't isolated those yet. I've deliberately been using the same prompt across the models to keep that part consistent, so I can't really say how much different prompts or expert routing patterns might affect performance.
The results definitely suggest that quantization and memory placement play a significant role, but I haven't measured memory bandwidth directly either, so I don't want to claim that's the whole explanation.
Testing different prompt types and lengths while tracking expert activation patterns would actually be a really interesting next step. You've given me another thing to add to my growing list of experiments.
Keeping the prompt constant was definitely the smart move for establishing a baseline. Varying prompt lengths while tracking expert activation is a brilliant next stepâI'd be fascinated to see if the routing patterns actually shift. Can't wait to read about those results if you post them!
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
three interleaved runs with medians is a real step up, but you also changed the quantization, the llama.cpp build and the offload settings in the same pass, so a flipped result could come from any of the three. i would pin the build first and rerun one model at both quants, since that isolates the quant effect cheaply.
on a laptop 4090 the thermal throttle also tends to show up in run three, not run one, so are you logging clocks alongside tokens per second?
Developer, maker, and perpetual project starter. I build web apps and dev tools mostly, things that I think are really useful and should exist and be simple to use. I also like to experiment with AI.
You're absolutely right about the multiple variables changing, and that's something I tried to be careful about in the follow-up conclusions. I actually did another round of 512-token tests using the same Qwen3.5 checkpoint at Q4_K_M and Q6_K, both on llama.cpp build 11985. Selective offloading was about 29.5% faster than traditional splitting at Q4, while the two were practically tied at Q6.
The offload configurations still needed adjusting for the different model sizes, though, so it's not a perfectly isolated quantization comparison yet.
And no, I haven't been logging GPU clocks alongside the throughput measurements. That's a really good suggestion, especially considering some of the variation I saw between runs. I'd like to capture clocks and temperatures alongside the actual output-token counts in the next round so I can better separate thermal effects from the offloading behavior.
Thank you for the feedback. You've given me quite a bit to investigate.
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
the q4 vs q6 split is useful. selective offload winning by 29.5% at q4 and tying at q6 reads like the gain shrinks once more of the model fits in vram. since the offload configs differ per model size, i would log how many layers sit on gpu for each run next to tokens per second, then the gap is explained by one number. when you add clocks and temps, will you log them per run or sample them every few seconds during decode?
Developer, maker, and perpetual project starter. I build web apps and dev tools mostly, things that I think are really useful and should exist and be simple to use. I also like to experiment with AI.
That's a good point about logging the GPU layer counts alongside throughput. I have the offload configurations recorded, but having the actual placement information right next to each result would make those comparisons much clearer.
And I was thinking of sampling clocks and temperatures every few seconds during decoding rather than just once per run. That way I could see whether performance changes line up with clock drops or temperature increases, especially during the longer generations.
I think adding that telemetry along with actual output-token counts would make the next round of tests a lot more useful. Thanks for all the suggestions. I'm learning quite a bit about benchmarking from these discussions alone.
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
sampling clocks every few seconds beats one reading per run, a throttle halfway through a 512 token decode would otherwise get averaged away into the final tok/s. i would log it on the same row as tok/s and layer counts so one csv answers whether a slow run was a placement problem or a thermal one. are you capturing cpu clocks too, since selective offload leans on the cpu side more?
The Gemma MTP result is the part I keep coming back to: an 11.68% draft acceptance rate turning a supposed speedup into a 46.7% slowdown (13.58 down to 7.24 tok/s) suggests acceptance rate should be reported before throughput in any speculative decoding benchmark. Did you notice whether acceptance changed between the shorter and the 512-token runs, or was it uniformly low? If it stays that low regardless of length, that points more to a drafter/target mismatch than to offloading overhead.
Developer, maker, and perpetual project starter. I build web apps and dev tools mostly, things that I think are really useful and should exist and be simple to use. I also like to experiment with AI.
Unfortunately, I only captured the acceptance telemetry during the longer 512-token test, so I don't have the numbers from the original 100-token runs to compare against.
I was pretty surprised by how low that 11.68% acceptance rate was, though. It definitely makes me wonder whether there's something going on with the drafter configuration or compatibility with the target model, rather than the slowdown being caused primarily by offloading overhead.
I'd like to repeat those tests while logging acceptance rates at both generation lengths, and ideally compare against a configuration without CPU offloading too. That should help narrow down what's actually causing the poor performance.
Thanks for bringing that up, I hadn't originally planned to investigate speculative decoding this deeply, but apparently I've found another rabbit hole.
Quite fascinating to see this! I understand you are perhaps limited by your hardware here and it costs when you go do these on real cloud infrastructure.
May I ask, are you trying specific kind of workloads and does the results vary? I understand these models are a bit generic but even then are you seeing specific differences? Just curious! Do you know if the models have published data on, is that SWE Bench, or wherever they do? I used to track them until 6 months or so ago
Developer, maker, and perpetual project starter. I build web apps and dev tools mostly, things that I think are really useful and should exist and be simple to use. I also like to experiment with AI.
I'm not really testing the models against specific tasks or evaluating their actual capabilities in these experiments. I'm more interested in the hardware side of things, specifically how different MoE architectures behave when they're too large to fit entirely in my GPU's VRAM.
I've been using the same prompt across all the models so I can compare generation speeds, prompt processing, and the differences between traditional layer offloading and selective expert offloading. And yes, even with a fairly generic workload, I've seen some pretty dramatic differences depending on the model, quantization, and offloading configuration.
The models do have their own published benchmarks for things like coding and reasoning, but I haven't been evaluating those here. I'm mostly just curious how much performance I can squeeze out of my 16GB GPU without relying on cloud infrastructure. It's turned into a much bigger experiment than I originally planned.
Thanks - now I get the intent, and that makes sense - "I'm mostly just curious how much performance I can squeeze out of my 16GB GPU without relying on cloud infrastructure." - I did this too for various LLM choices with OMLX (not MOE) to know where I can use local LLM and save money.
"It's turned into a much bigger experiment than I originally planned." - this is what I discovered and had to scale back tests and ended up looking for other people's benchmarks for my use cases to pair down the LLM choices.
The confound you flag at the end is the real story: quant, offload config, and llama.cpp build all changed at once, so the Qwen and LFM2.5 flips can't be attributed to quantization alone. Isolating it needs the two-quant A/B (Q4_K_M vs Q6_K) on all five models with everything else pinned - you only ran that pair on Qwen. Also, with --n-cpu-moe the payoff depends on how skewed per-token expert activation is. Did you log distinct experts activated per token? Gemma tying at 13.58 tok/s both ways smells like uniform expert usage - nothing for selective scheduling to exploit. A histogram of expert hits would explain the tie and predict which architectures benefit before you benchmark.
Developer, maker, and perpetual project starter. I build web apps and dev tools mostly, things that I think are really useful and should exist and be simple to use. I also like to experiment with AI.
You're right that changing multiple variables makes it difficult to isolate exactly what caused the performance differences. That's why I was careful to point that out in the follow-up.
I did run an additional Q4_K_M vs. Q6_K comparison on the same Qwen checkpoint using the same llama.cpp build and longer 512-token generations, which helped narrow things down a bit. The offload configurations still differed, though, so there's definitely more to investigate.
I haven't logged individual expert activation frequencies yet, but that's actually a really interesting idea. I'd be curious to see whether certain experts are consistently selected more often and how that relates to offloading performance.
I'm not sure Gemma's tie necessarily means uniform expert usage, since there could be several reasons the two configurations performed similarly. But looking at the routing distribution might help answer some of those questions.
I research AI tools, APIs & agents â comparing specs, pricing and real user reports â then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
Redoing it all at Q4_K_M and interleaving the runs is the honest fix â the first version's mixed quantizations made it impossible to tell what was architecture and what was just precision. The LFM2.5 swing is what I'd want to see next though: did it actually get faster once the quantization was consistent, or is selective offloading just a bad fit for that one regardless?
Developer, maker, and perpetual project starter. I build web apps and dev tools mostly, things that I think are really useful and should exist and be simple to use. I also like to experiment with AI.
That LFM result was actually one of the biggest surprises for me. At Q8_0, selective offloading was only getting 2.11 tok/s compared to 9.03 with traditional splitting. But when I switched to Q4_K_M, the initial runs jumped to 35.44 tok/s with selective offloading.
I did some longer 512-token tests afterward, and both strategies settled around 22 tok/s, so they were practically tied. That definitely suggests selective offloading isn't inherently a bad fit for LFM.
I can't attribute the improvement entirely to quantization since I also adjusted the offload configuration and updated llama.cpp between tests. But it's a pretty dramatic difference, and I'd love to isolate those variables further. Apparently this rabbit hole isn't finished with me yet.
Retesting until the conclusion changes is exactly the evidence developers need to see. The reusable artifact would be a benchmark manifest that locks model revision, routing policy, host topology, warm-up, workload mix, and cost accounting; otherwise an offloading result can drift while every individual run still looks plausible.
Developer, maker, and perpetual project starter. I build web apps and dev tools mostly, things that I think are really useful and should exist and be simple to use. I also like to experiment with AI.
That's a really good suggestion. One thing these experiments have definitely taught me is just how many variables can affect the results, even when you think you're keeping things consistent. I started out just wanting to compare two offloading methods, and ended up learning quite a bit about benchmarking itself along the way.
I've been keeping detailed experiment logs, but a standardized manifest that captures the exact model, configuration, hardware, and workload for each run would make comparisons and reproduction much easier. Honestly, that sounds like a pretty useful little tool to build too, and now you've got me thinking.
I write when an idea won't leave me alone đ§ Building AI agents and the tools to build AI agents. Love connecting AI with other fields and yapping about all of them.
The 11.68% acceptance came from one prose prompt about Arch Linux, and I'd guess the text type matters a lot there. A drafter gets further ahead on code or JSON, where the next tokens are easy to predict. One Gemma MTP run on a code prompt could tell you whether the drafter is mismatched or the prose was hard to guess.
Retesting after reader feedback is exactly the right instinct, and the two fixes you call out â consistent quantization and repeated runs â are the same controls we reach for in Electron performance work: a single cold-start sample is noise, and comparing two builds with different V8 flags or resource limits makes the delta meaningless. The lesson that 'which offloading strategy wins depends on the model' maps well to desktop: there is no universal best process split, only the one that holds under your actual workload distribution. Publishing the correction explicitly is also a service â benchmarks that silently change under the hood are worse than no benchmark.
A technologist working in automation. Here I write up what I actually learn building tools, engineering AI agents, and keeping systems running, along with the workflows and habits that stick.
The Q4 versus Q6 swing on selective offloading makes sense when you look at host memory bus throughput. With selective offloading, the attention layers stay in VRAM and the CPU only streams the active routed experts from system RAM. On a dual-channel DDR5 laptop setup you usually tap out around 70 GB/s of actual system memory bandwidth. At Q6_K, fetching the active expert tensors pushes right up against that memory bus limit, which stalls CPU execution on every token. Dropping to Q4_K_M cuts the weight transfer volume by roughly a third, keeping the active expert streaming comfortably under the bus saturation point. Traditional layer splitting does not see that same bump because it pays a full PCIe synchronization barrier on every layer boundary regardless of expert sparsity.
Developer, maker, and perpetual project starter. I build web apps and dev tools mostly, things that I think are really useful and should exist and be simple to use. I also like to experiment with AI.
That's a really interesting explanation, and it definitely fits with what I was seeing in the Qwen tests. The difference between Q4_K_M and Q6_K was pretty significant, especially with selective offloading.
I haven't actually measured the memory bandwidth during inference yet, so I can't say for sure where it's hitting a limit, but that's something I'd really like to investigate next. It would be interesting to see whether the bandwidth usage lines up with the performance changes or whether there's more going on with CPU execution and synchronization.
Thanks for the detailed explanation! Every time I think I'm getting to the bottom of this, I find something else worth testing.
Implementation Specialist and Database Developer with over 15 years of experience in the IT industry, specializing in the design, implementation and management of relational databases.
Developer, maker, and perpetual project starter. I build web apps and dev tools mostly, things that I think are really useful and should exist and be simple to use. I also like to experiment with AI.
I think selective MoE offloading could definitely be useful on an 8GB GPU too, but I wouldn't assume the same performance differences would hold. With less VRAM, you'd need to keep more of the model in system RAM, which could shift the balance between traditional and selective offloading quite a bit.
I wouldn't necessarily call 16GB a sweet spot either. It's just what I happen to have available for testing. System RAM, memory bandwidth, quantization, and the specific model architecture all play a role.
I'd actually be really interested in seeing these experiments on an 8GB GPU to see how the results compare.
We're a place where coders share, stay up-to-date and grow their careers.
One thing I'd be curious to see in the next round is how context depth changes the offloading results. A setup that looks fine at 512 tokens might behave very differently once the KV cache grows and starts competing for VRAM with the model weights.
I'd run a few
llama-benchtests with fixed generation lengths but different context depths, using-dand-pgto separate prompt processing from generation. That could help distinguish whether the slowdown is coming from expert weight transfers, attention/KV-cache pressure, or the interaction between them.Especially with LFM, where the 100-token and 512-token results told quite different stories, I think that would be a useful next experiment.
I actually did some context scaling tests with Phi-3.5 and Apex in my original experiments, going from 4K to 16K, but those were changes to the configured context window rather than tests with prompts that actually filled it.
I haven't explored how growing the actual context affects the different offloading strategies yet, and I think that would be really interesting, especially with the differences I saw between the shorter and longer LFM runs.
Using llama-bench to separate prompt processing and generation at different context depths sounds like a good next step. I'd love to see whether the KV cache starts affecting which offloading configuration performs better as the context grows.
Thanks for the suggestion! My list of things to test keeps getting longer.
That distinction between configured context size and actually filling the context is important. A 16K context window doesn't necessarily tell us much about memory pressure if the prompt only uses a fraction of it.
I'd be particularly curious about what happens when the KV cache starts competing with expert weights for GPU memory. In an MoE setup, that could change the offloading trade-off quite noticeably, especially if some experts are already being moved between CPU and GPU during generation.
It'd be interesting to see whether the crossover point between different offloading strategies shifts as the actual context grows. That might explain some of the differences you observed in the LFM runs.
Seeing someone revisit their own conclusions publicly is interesting to me, especially when the newer results complicate the original explanation. That part of research doesn't get discussed enough.
I think it's easy to get attached to an explanation once you've written it all out, but sometimes the more you investigate, the more complicated things get. I actually enjoyed discovering that some of my original assumptions weren't quite right. It gave me a lot more to explore, and I learned more from going back and questioning the results than I probably would have otherwise.
Your reply reminds me of that old Bertrand Russell observation about intelligent people being full of doubt. Iâve always found that interesting because the deeper you investigate something, the more you realize how much you might be missing. Being willing to question your own conclusions seems like one of the most important parts of research.
The early-ending Apex run makes me want actual output-token counts beside each timing. Were the 512-token tests all reaching 512, or was that just the limit? That would help tell whether the longer runs are measuring comparable amounts of work before digging into why they slowed down.
That's a really good point, and something I should have documented more carefully. The 512 tokens was the configured generation limit, but I didn't include the actual output-token counts for each run in my results.
The early-ending Apex run made me realize how much that can affect a short benchmark, and you're right that it's important to verify the longer runs were actually doing comparable amounts of work too.
I think my next step is going to be logging the actual generated token counts alongside the timing data, rather than relying on the configured limit. Thanks for pointing that out.
The 'a measurement and an explanation aren't the same thing' is the most important line in this article. So many benchmark write-ups stop at the number and never dig into why it happened. The fact that you went back and re-tested after feedback makes this much more credible.
Thank you! That's actually one of the biggest things I've taken away from all this. It's one thing to observe a performance difference, but understanding exactly what's causing it is a whole different challenge. And I've definitely caught myself coming up with explanations that sounded perfectly reasonable until I did more testing. I'm glad I went back and revisited the results. It's been a really interesting learning experience.
This level of intellectual honesty is rare to see in benchmark posts. It takes a lot of discipline to go back, rerun the tests, and publicly admit that the initial "perfectly reasonable" narrative was wrong. It's a great reminder that performance metrics are just numbers until you actually understand the mechanism behind them.
I'm curious â when you were digging into the data this time around, did you find that the MoE offloading behavior was more dependent on the specific prompt structure, or was it mostly the hardware memory bandwidth limits doing the heavy lifting?
Thank you. I really appreciate that. I think one of the most interesting parts of experimenting with things like this is discovering that an explanation you were pretty confident about doesn't necessarily hold up once you start testing further.
As for the prompt structure versus memory bandwidth, I haven't isolated those yet. I've deliberately been using the same prompt across the models to keep that part consistent, so I can't really say how much different prompts or expert routing patterns might affect performance.
The results definitely suggest that quantization and memory placement play a significant role, but I haven't measured memory bandwidth directly either, so I don't want to claim that's the whole explanation.
Testing different prompt types and lengths while tracking expert activation patterns would actually be a really interesting next step. You've given me another thing to add to my growing list of experiments.
Keeping the prompt constant was definitely the smart move for establishing a baseline. Varying prompt lengths while tracking expert activation is a brilliant next stepâI'd be fascinated to see if the routing patterns actually shift. Can't wait to read about those results if you post them!
three interleaved runs with medians is a real step up, but you also changed the quantization, the llama.cpp build and the offload settings in the same pass, so a flipped result could come from any of the three. i would pin the build first and rerun one model at both quants, since that isolates the quant effect cheaply.
on a laptop 4090 the thermal throttle also tends to show up in run three, not run one, so are you logging clocks alongside tokens per second?
You're absolutely right about the multiple variables changing, and that's something I tried to be careful about in the follow-up conclusions. I actually did another round of 512-token tests using the same Qwen3.5 checkpoint at Q4_K_M and Q6_K, both on llama.cpp build 11985. Selective offloading was about 29.5% faster than traditional splitting at Q4, while the two were practically tied at Q6.
The offload configurations still needed adjusting for the different model sizes, though, so it's not a perfectly isolated quantization comparison yet.
And no, I haven't been logging GPU clocks alongside the throughput measurements. That's a really good suggestion, especially considering some of the variation I saw between runs. I'd like to capture clocks and temperatures alongside the actual output-token counts in the next round so I can better separate thermal effects from the offloading behavior.
Thank you for the feedback. You've given me quite a bit to investigate.
the q4 vs q6 split is useful. selective offload winning by 29.5% at q4 and tying at q6 reads like the gain shrinks once more of the model fits in vram. since the offload configs differ per model size, i would log how many layers sit on gpu for each run next to tokens per second, then the gap is explained by one number. when you add clocks and temps, will you log them per run or sample them every few seconds during decode?
That's a good point about logging the GPU layer counts alongside throughput. I have the offload configurations recorded, but having the actual placement information right next to each result would make those comparisons much clearer.
And I was thinking of sampling clocks and temperatures every few seconds during decoding rather than just once per run. That way I could see whether performance changes line up with clock drops or temperature increases, especially during the longer generations.
I think adding that telemetry along with actual output-token counts would make the next round of tests a lot more useful. Thanks for all the suggestions. I'm learning quite a bit about benchmarking from these discussions alone.
sampling clocks every few seconds beats one reading per run, a throttle halfway through a 512 token decode would otherwise get averaged away into the final tok/s. i would log it on the same row as tok/s and layer counts so one csv answers whether a slow run was a placement problem or a thermal one. are you capturing cpu clocks too, since selective offload leans on the cpu side more?
The Gemma MTP result is the part I keep coming back to: an 11.68% draft acceptance rate turning a supposed speedup into a 46.7% slowdown (13.58 down to 7.24 tok/s) suggests acceptance rate should be reported before throughput in any speculative decoding benchmark. Did you notice whether acceptance changed between the shorter and the 512-token runs, or was it uniformly low? If it stays that low regardless of length, that points more to a drafter/target mismatch than to offloading overhead.
Unfortunately, I only captured the acceptance telemetry during the longer 512-token test, so I don't have the numbers from the original 100-token runs to compare against.
I was pretty surprised by how low that 11.68% acceptance rate was, though. It definitely makes me wonder whether there's something going on with the drafter configuration or compatibility with the target model, rather than the slowdown being caused primarily by offloading overhead.
I'd like to repeat those tests while logging acceptance rates at both generation lengths, and ideally compare against a configuration without CPU offloading too. That should help narrow down what's actually causing the poor performance.
Thanks for bringing that up, I hadn't originally planned to investigate speculative decoding this deeply, but apparently I've found another rabbit hole.
Quite fascinating to see this! I understand you are perhaps limited by your hardware here and it costs when you go do these on real cloud infrastructure.
May I ask, are you trying specific kind of workloads and does the results vary? I understand these models are a bit generic but even then are you seeing specific differences? Just curious! Do you know if the models have published data on, is that SWE Bench, or wherever they do? I used to track them until 6 months or so ago
Apologies, I have not read all your posts.
I'm not really testing the models against specific tasks or evaluating their actual capabilities in these experiments. I'm more interested in the hardware side of things, specifically how different MoE architectures behave when they're too large to fit entirely in my GPU's VRAM.
I've been using the same prompt across all the models so I can compare generation speeds, prompt processing, and the differences between traditional layer offloading and selective expert offloading. And yes, even with a fairly generic workload, I've seen some pretty dramatic differences depending on the model, quantization, and offloading configuration.
The models do have their own published benchmarks for things like coding and reasoning, but I haven't been evaluating those here. I'm mostly just curious how much performance I can squeeze out of my 16GB GPU without relying on cloud infrastructure. It's turned into a much bigger experiment than I originally planned.
Thanks - now I get the intent, and that makes sense - "I'm mostly just curious how much performance I can squeeze out of my 16GB GPU without relying on cloud infrastructure." - I did this too for various LLM choices with OMLX (not MOE) to know where I can use local LLM and save money.
"It's turned into a much bigger experiment than I originally planned." - this is what I discovered and had to scale back tests and ended up looking for other people's benchmarks for my use cases to pair down the LLM choices.
The confound you flag at the end is the real story: quant, offload config, and llama.cpp build all changed at once, so the Qwen and LFM2.5 flips can't be attributed to quantization alone. Isolating it needs the two-quant A/B (Q4_K_M vs Q6_K) on all five models with everything else pinned - you only ran that pair on Qwen. Also, with --n-cpu-moe the payoff depends on how skewed per-token expert activation is. Did you log distinct experts activated per token? Gemma tying at 13.58 tok/s both ways smells like uniform expert usage - nothing for selective scheduling to exploit. A histogram of expert hits would explain the tie and predict which architectures benefit before you benchmark.
You're right that changing multiple variables makes it difficult to isolate exactly what caused the performance differences. That's why I was careful to point that out in the follow-up.
I did run an additional Q4_K_M vs. Q6_K comparison on the same Qwen checkpoint using the same llama.cpp build and longer 512-token generations, which helped narrow things down a bit. The offload configurations still differed, though, so there's definitely more to investigate.
I haven't logged individual expert activation frequencies yet, but that's actually a really interesting idea. I'd be curious to see whether certain experts are consistently selected more often and how that relates to offloading performance.
I'm not sure Gemma's tie necessarily means uniform expert usage, since there could be several reasons the two configurations performed similarly. But looking at the routing distribution might help answer some of those questions.
Thanks for the suggestions.
Redoing it all at Q4_K_M and interleaving the runs is the honest fix â the first version's mixed quantizations made it impossible to tell what was architecture and what was just precision. The LFM2.5 swing is what I'd want to see next though: did it actually get faster once the quantization was consistent, or is selective offloading just a bad fit for that one regardless?
That LFM result was actually one of the biggest surprises for me. At Q8_0, selective offloading was only getting 2.11 tok/s compared to 9.03 with traditional splitting. But when I switched to Q4_K_M, the initial runs jumped to 35.44 tok/s with selective offloading.
I did some longer 512-token tests afterward, and both strategies settled around 22 tok/s, so they were practically tied. That definitely suggests selective offloading isn't inherently a bad fit for LFM.
I can't attribute the improvement entirely to quantization since I also adjusted the offload configuration and updated llama.cpp between tests. But it's a pretty dramatic difference, and I'd love to isolate those variables further. Apparently this rabbit hole isn't finished with me yet.
Retesting until the conclusion changes is exactly the evidence developers need to see. The reusable artifact would be a benchmark manifest that locks model revision, routing policy, host topology, warm-up, workload mix, and cost accounting; otherwise an offloading result can drift while every individual run still looks plausible.
That's a really good suggestion. One thing these experiments have definitely taught me is just how many variables can affect the results, even when you think you're keeping things consistent. I started out just wanting to compare two offloading methods, and ended up learning quite a bit about benchmarking itself along the way.
I've been keeping detailed experiment logs, but a standardized manifest that captures the exact model, configuration, hardware, and workload for each run would make comparisons and reproduction much easier. Honestly, that sounds like a pretty useful little tool to build too, and now you've got me thinking.
Thank you. I really appreciate the feedback.
The 11.68% acceptance came from one prose prompt about Arch Linux, and I'd guess the text type matters a lot there. A drafter gets further ahead on code or JSON, where the next tokens are easy to predict. One Gemma MTP run on a code prompt could tell you whether the drafter is mismatched or the prose was hard to guess.
Retesting after reader feedback is exactly the right instinct, and the two fixes you call out â consistent quantization and repeated runs â are the same controls we reach for in Electron performance work: a single cold-start sample is noise, and comparing two builds with different V8 flags or resource limits makes the delta meaningless. The lesson that 'which offloading strategy wins depends on the model' maps well to desktop: there is no universal best process split, only the one that holds under your actual workload distribution. Publishing the correction explicitly is also a service â benchmarks that silently change under the hood are worse than no benchmark.
The Q4 versus Q6 swing on selective offloading makes sense when you look at host memory bus throughput. With selective offloading, the attention layers stay in VRAM and the CPU only streams the active routed experts from system RAM. On a dual-channel DDR5 laptop setup you usually tap out around 70 GB/s of actual system memory bandwidth. At Q6_K, fetching the active expert tensors pushes right up against that memory bus limit, which stalls CPU execution on every token. Dropping to Q4_K_M cuts the weight transfer volume by roughly a third, keeping the active expert streaming comfortably under the bus saturation point. Traditional layer splitting does not see that same bump because it pays a full PCIe synchronization barrier on every layer boundary regardless of expert sparsity.
That's a really interesting explanation, and it definitely fits with what I was seeing in the Qwen tests. The difference between Q4_K_M and Q6_K was pretty significant, especially with selective offloading.
I haven't actually measured the memory bandwidth during inference yet, so I can't say for sure where it's hitting a limit, but that's something I'd really like to investigate next. It would be interesting to see whether the bandwidth usage lines up with the performance changes or whether there's more going on with CPU execution and synchronization.
Thanks for the detailed explanation! Every time I think I'm getting to the bottom of this, I find something else worth testing.
Hi :-) would these conclusions hold on a smaller 8GB GPU, or is 16GB a sweet spot?
I think selective MoE offloading could definitely be useful on an 8GB GPU too, but I wouldn't assume the same performance differences would hold. With less VRAM, you'd need to keep more of the model in system RAM, which could shift the balance between traditional and selective offloading quite a bit.
I wouldn't necessarily call 16GB a sweet spot either. It's just what I happen to have available for testing. System RAM, memory bandwidth, quantization, and the specific model architecture all play a role.
I'd actually be really interested in seeing these experiments on an 8GB GPU to see how the results compare.