DEV Community

Cover image for Thinking On or Off? Gemma 4 E4B on an 8 GB RTX 3050
xbill Subscriber for Google Developer Experts

Posted on

Thinking On or Off? Gemma 4 E4B on an 8 GB RTX 3050

This article provides a step by step guide to measuring what thinking buys Gemma 4 E4B on a local GPU, and what it costs in time. E4B runs on an 8 GB RTX 3050 through llama.cpp and answers every GSM8K test problem and all 30 AIME 2025 problems with thinking off and with thinking on, on two 4-bit builds of the same QAT weights, each with and without a drafter. It is the second model in the series, after Gemma 4 E2B, on the same card, the same problems and the same settings.

https://github.com/xbill9/gemma4-dev/tree/main/docs/gemma4-e4b-thinking-3050

The short version: E4B gets 93 to 94 percent of GSM8K right. Thinking more than doubled the time and lifted the score by at most 16 problems. The drafter made every run up to 2.39 times faster and changed no score.


The Findings

How E4B scores. GSM8K 0.9272 to 0.9439 across the eight setups, and 9 to 13 of 30 on AIME 2025.

What thinking cost. On GSM8K, thinking on produced 2.24 to 2.30 times as many tokens per problem and took 2.24 to 2.47 times as long.

What thinking bought. A little, at most. GSM8K rose in all four setups, by +2, +4, +13 and +16 problems. The +16, Google's file with the drafter, passes a paired test at p = 0.0440; the other three do not (p = 0.1175 to 0.8919). With four tests, one passing at 0.05 happens by chance alone 18.5 percent of the time. AIME moved by −4 to +2 problems, within chance.

What the drafter bought. Every GSM8K run finished 1.99 to 2.39 times sooner with Google's QAT drafter, at 130.8 to 132.3 against 57.0 to 58.4 generated tokens per second with the repack. It changed no score beyond chance in any of eight paired tests.

The two weight builds. The exact Q4_0 repack and Google's QAT Q4_0 file scored level on every paired test. The repack is 4.22 GB against 5.15 GB, and serves in 6,430 MiB against 7,326 MiB.

What to do with it.

  • 🟢 Serve E4B with the drafter. It is the one change here that pays every time.
  • ⚠️ Thinking on adds about a point on GSM8K at best, for more than twice the time.

Caveats. One run of each setting, temperature 0. AIME's 30 problems can only show large effects.


What Was Measured

The E2B article set up the question: on a local GPU the bill for thinking is time, and Gemma 4's chat template has two thinking settings, off and on. This run repeats it with the next model up.

One model, Gemma 4 E4B, on one card, with three factors crossed:

Factor Settings
Thinking off, on (enable_thinking in the chat template)
Weights exact Q4_0 repack of Google's QAT weights, Google's QAT Q4_0 GGUF
Drafter none, Google's E4B QAT assistant as exact Q4_0, 2 drafted tokens
Problems GSM8K test, 1,319; AIME 2025 I and II, 30

That is 16 runs, 26.83 hours of GPU time, every reply saved.


At This Point You Should Have…

  • An NVIDIA card with 8 GB, here an RTX 3050 OEM on driver 615.78.08
  • llama.cpp built with CUDA, here commit 7182ef71e
  • Python 3 with huggingface_hub, tokenizers and scipy
  • A clone of gemma4-dev

Step 1 — Two Builds of the Same Weights, and a Drafter

Google trained Gemma 4 E4B to run at 4 bits (quantization-aware training, QAT) and ships it as a Q4_0 GGUF. The repack, xbill9/gemma-4-E4B-it-qat-q4_0-exact-gguf, stores the same QAT 4-bit values in a smaller file.

File Bytes Card memory when serving
Repack, gemma-4-E4B-it-q4_0-exact.gguf 4,215,695,712 6,430 MiB
Google, gemma-4-E4B_q4_0-it.gguf 5,154,941,280 7,326 MiB
Drafter, gemma-4-E4B-it-qat-assistant-q4_0-exact.gguf 59,678,368 +194 MiB

The drafter is Google's small E4B QAT assistant model, stored as exact Q4_0. It guesses the next two tokens and the main model checks them in one step, so accepted guesses come out for the price of one step.

Every run serves the same way: the full 131,072-token context with an f16 cache, the per-layer embedding table on the card, one request at a time, temperature 0, and a 32,768-token limit on each reply, thinking included. Google's file with the drafter uses 7,520 of the card's 8,192 MiB.


Step 2 — Run All Sixteen

cd local-llamacpp-3050oem-4b-q4_0/benchmarks/runs/2026-11-17-thinking-e4b-3050oem
setsid nohup ./chain.sh > nohup-chain.out 2>&1 < /dev/null &
Enter fullscreen mode Exit fullscreen mode

run.sh reads the model's files from wave.env, starts the server once per weights and drafter pair, and runs four cells on it: GSM8K thinking off, AIME off, AIME on, GSM8K on. The scoring script asks for the final answer on its own last line as Answer: <number> and marks a reply right when that number matches. Each reply is saved as it finishes, so a stopped run picks up where it left off.

The run's own log, driver.log, shows one line per cell:

2026-11-17 12:58:18 59, 1897 MHz END gsm8k repack-none thinking=off rc=0 {"task": "gsm8k", "max_tokens": 32768, "n": 1319, "right": 1226, "accuracy": 0.9295, "errors": 0, "truncated": 0, ...
Enter fullscreen mode Exit fullscreen mode

Step 3 — The Scores

analyze.py builds every table from the saved replies.

GSM8K, 1,319 problems

Weights Drafter Thinking off Thinking on
Repack none 0.9295 0.9325
Repack yes 0.9272 0.9371
Google none 0.9348 0.9363
Google yes 0.9318 🥇 0.9439

All eight sit between 0.9272 and 0.9439, and every 95% range overlaps every other.

AIME 2025, 30 problems, right answers

Weights Drafter Thinking off Thinking on
Repack none 12 10
Repack yes 10 12
Google none 12 12
Google yes 🥇 13 9

No reply in any of the 16 runs reached the 32,768-token limit.


Step 4 — Where the Tokens Go

tokens.py counts thinking and answer tokens in each saved reply with Gemma 4 E4B's tokenizer. Its count matches the server's on 5,393 of 5,396 replies at each thinking setting, up to the fixed end-of-turn and thinking markers, and is one token off on the other 3.

GSM8K off GSM8K on AIME off AIME on
Thinking tokens, mean 0 505 to 514 0 3,760 to 4,288
Answer tokens, mean 340 to 344 263 to 264 3,582 to 4,174 1,138 to 1,336

On GSM8K the answer shrinks by about a quarter and about 510 thinking tokens are added in front of it. With thinking off on AIME, E4B works the problem out in the answer itself; with thinking on, that working moves into the thinking text.

Thinking on used more tokens on 1,309 to 1,313 of the 1,319 GSM8K problems and on 23 to 26 of 30 AIME problems in every setup, far beyond chance (signed-rank p below 0.006 in all eight).


Step 5 — Time and the Drafter

Weights Drafter GSM8K off GSM8K on AIME off AIME on
Repack none 2.38 h 5.45 h 0.52 h 0.76 h
Repack yes 🥇 1.00 h 2.46 h 0.38 h 🥇 0.34 h
Google none 2.34 h 5.24 h 0.62 h 0.80 h
Google yes 1.16 h 2.64 h 0.33 h 0.40 h

The drafter made every GSM8K run 1.99 to 2.39 times faster and every AIME run 1.38 to 2.23 times faster. The main model accepted 0.83 to 0.84 of the drafted tokens on GSM8K and 0.80 to 0.82 on AIME, and generation rose from 54.5–58.4 tokens per second to 121.6–132.3.

Thinking took 1.19 to 1.46 times as long on AIME in three setups. In the fourth, the repack with the drafter, the thinking-off run spent 395 seconds outside generation on a few replies, so its clock time reads 0.90 times; by generation time alone thinking took 1.21 times as long.

Per problem, GSM8K took 2.72 seconds with the repack, the drafter and thinking off, and 14.87 seconds with the repack, no drafter and thinking on.


Step 6 — Is Any Difference Bigger Than Chance?

The same problems ran in every cell, so each comparison is an exact paired test on the problems one setting got right and the other got wrong.

Comparison, GSM8K Right only before Right only after p
Thinking on, repack, no drafter 25 29 0.6835
Thinking on, repack, drafter 23 36 0.1175
Thinking on, Google, no drafter 26 28 0.8919
Thinking on, Google, drafter 20 36 0.0440
Drafter, largest change 10 20 0.0987
Google instead of repack, largest change 15 24 0.1996

Thinking changed the answer on 54 to 59 GSM8K problems in each setup, and the gains outnumber the losses in all four, by 2 to 16. On AIME no comparison does better than p = 0.2891. The full table of 24 tests is in RESULTS.md.


🔎 Tip: Turn the Drafter On Before You Turn Thinking On

Both change how long an answer takes. On E4B the drafter cut the time by 1.38 to 2.39 times and changed no answer beyond chance; thinking cost 2.24 to 2.47 times the time on GSM8K for 2 to 16 more problems. With both on, a GSM8K problem took 6.72 to 7.20 seconds, against 14.30 to 14.87 for thinking on without the drafter.


Compare and Contrast

Thinking off Thinking on
GSM8K score range 0.9272 to 0.9348 0.9325 to 🥇 0.9439
AIME right, of 30 10 to 13 9 to 12
GSM8K time, with drafter 🥇 1.00 to 1.16 h 2.46 to 2.64 h
Thinking tokens per GSM8K problem 🥇 0 505 to 514
Score change beyond chance — one of eight tests

So, Which One?

For math on an 8 GB card, serve E4B with the drafter: a full GSM8K pass takes 1.00 hour with the repack and thinking off, against 2.38 hours without the drafter.

Leave thinking off for grade-school math unless the last point matters. It raised the score in all four setups, by 2 to 16 problems, one of them beyond chance, and it more than doubled the time.

On hard math, thinking on gave E4B no extra AIME answers: 9 to 12 right against 10 to 13 off.

Between the two 4-bit builds, take the repack: same answers, 0.94 GB smaller and 896 MiB less card memory.


Summary

The goal of this article was to measure what thinking buys Gemma 4 E4B on a local GPU and what it costs in time. The key to the solution was holding the model and card fixed and crossing thinking with the weight build and the drafter, on the same 1,349 problems in every cell. The results were:

  • 🟢 E4B scored 0.9272 to 0.9439 on GSM8K and 9 to 13 of 30 on AIME 2025.
  • ⚠️ Thinking took 2.24 to 2.47 times as long on GSM8K for 2 to 16 more problems, one of four gains beyond chance.
  • ❌ Thinking gave no AIME gain: −4 to +2 problems.
  • 🟢 The drafter made every GSM8K run 1.99 to 2.39 times faster and changed no score beyond chance.
  • 🟢 The exact Q4_0 repack and Google's QAT file scored level; the repack is 0.94 GB smaller.
  • 🟢 No reply hit the 32,768-token limit.

Scope: one RTX 3050 OEM with 8 GB in one desktop, llama.cpp 7182ef71e, Gemma 4 E4B, one run of each of 16 settings on 2026-11-17 and 18, temperature 0, one request at a time, the full 131,072-token context, a 32,768-token reply limit. GSM8K is all 1,319 test problems and AIME 2025 is papers I and II. Times are for this card and move with clock and temperature; the 12B is measured separately.

The strategy for measuring thinking against a drafter on a local GPU was validated with an incremental step by step approach.


References

Hope you enjoyed it!

Top comments (1)

Collapse
 
gautamtalksdev profile image
Gautam K •

Testing thinking on and off on an 8 GB card is useful, since that's what a lot of people actually own. Did thinking mode change answer quality enough to be worth the extra tokens on that hardware?