4-bit Naive Bayes: symmetric fails on AG News, zero-point rescues it
Minimum bits is not one knob. A small model has two budgets: data bytes and weight
bits. I ran two experiments on the same project to see which one matters where.
1. 4-bit Naive Bayes
The SMS model is a 160-word Complement NB with int8 weights. I quantized the same weights to
int4 with a per-class scale and kept the raw integer dot product (the rule the fixed SMS
containers deploy). Five 80/20 seeds:
| precision | weight bytes | mean accuracy |
|---|---|---|
| float | 1,280 B | 91.6% |
| int8 | 320 B | 94.2% |
| int4 | 160 B | 93.2% |
On SMS, int4 keeps 93.2% at half the weight bytes. Seed 0 is 93.6% int4 vs 93.5% int8.
The full JSON still holds the 160-word vocabulary (about 1.2 KB), so the artifact shrinks by
roughly 10%, not 50%. The weight matrix itself is halved.
Then the same recipe on the 500-example, 4-class AG News coreset:
| precision | weight bytes | accuracy |
|---|---|---|
| int8 | 20,596 B | 78.3% |
| int4 | 10,298 B | 66.5% |
It collapses. Four classes and a 5,149-word vocabulary do not fit into 15 signed levels
when all log-probability weights are negative.
A zero-point (affine) int4 uses all 16 codes and recovers most of it:
| AG News precision | weight bytes | accuracy |
|---|---|---|
| int8 | 20,596 B | 78.3% |
| symmetric int4 | 10,298 B | 66.5% |
| zero-point int4 | 10,298 B | 76.8% |
That is only 1.5 pp below int8 at half the weight bytes. On SMS the trade goes the other
way: symmetric int4 is 93.2%, affine int4 is 91.8%. The binary task prefers symmetric; the
four-class task needs the zero-point.
2. Same bytes, better examples?
The 130 KB AG News demo uses the first N rows per class. I compared that with random,
token coverage, TF-IDF k-means diversity and uncertainty sampling at the same example count
(400-row test):
| rows/class | first | random | coverage | TF-IDF | uncertainty |
|---|---|---|---|---|---|
| 50 | 66.0% / 52,307 B | 69.8 ± 2.4% | 62.5% / 75,987 B | 72.8% / 54,320 B | 63.0% |
| 125 | 78.2% / 132,798 B | 74.6 ± 2.4% | 71.2% / 162,774 B | 72.5% | 74.8% |
| 250 | 81.8% / 266,331 B | 76.2 ± 0.7% | 76.5% / 298,570 B | 74.8% | 80.0% |
At 50 rows/class, TF-IDF diversity buys +6.8 pp over first-N for about 2 KB more.
At 125 and 250 rows/class, first-N is the strongest baseline. The current 130 KB
artifact is already in the regime where selection tricks do not help.
Coverage and uncertainty select longer/noisier rows and spend more bytes; they lose here.
The interesting rule is budget-dependent: diversity at very small budgets, first-N once the
budget is large enough.
What this means
- Data bytes and weight bits are separate budgets. At 50 examples/class, better selection can beat more bits; at 125+, first-N is already strong.
- 4-bit weights are scheme-dependent. Symmetric int4 survives on binary SMS but loses 12 points on 4-class AG News; a zero-point int4 recovers AG News to within 1.5 points of int8 at half the weight bytes.
- The fixed 93.5% SMS artifact is the raw-int8 rule. The per-class scale is quantization
metadata, not part of inference; applying it changed the argmax and lowered the score.
tests/test_sms_rule.pypins this.
What I would try next
- Per-feature scales or a bias correction for multiclass int4, instead of one scale per class.
- A budget-aware selector that switches to TF-IDF diversity below ~50 examples/class.
- A cross-task sweep of symmetric vs zero-point int4 on more than two datasets.
- Cross-validation on the selection strategies; these numbers are one 400-row test.
Everything is one machine, one teacher-labelled AG News sample. The negative result is the
point: minimum bits is not a single knob, and 4-bit is not automatically free.
- Repo: https://github.com/byjameson/stream-finetune
- 130 KB AI: https://byjameson-mini-ai.static.hf.space/mini-ai.html
- Maker (5-fold CV / calibration): https://byjameson-mini-ai.static.hf.space/maker.html
- Code:
research/int4_nb.py,research/int4_affine.py,research/coreset_selection.py
Top comments (5)
That contrast is more informative than a single accuracy result because it exposes the boundary of the approximation. A useful next measurement would be calibration and error slices by class frequency or document length; that can show whether the failure is quantization noise, a feature-distribution shift, or simply a dataset whose signal needs more capacity.
Good next step. Current split: SMS int8 94.2% vs symmetric int4 93.2% over 5 seeds; the per-seed delta runs from +0.1 to -2.8 pp, which looks like quantization noise plus a few strong tokens. AG News: symmetric int4 66.5%, zero-point int4 76.8%, int8 78.3% — the grid was the first failure. Calibration on the fixed 1,579 B SMS model is Brier 0.144 / ECE 0.066; a nested T=1.5 lowers ECE to 0.032. I have not run calibration for the int4 variants yet. Class-frequency and length slices are the next measurement; my hypothesis is that AG int4 failure concentrates in Sci/Tech (the smallest class) and long documents, but that is unmeasured.
Those slices should tell you whether the affine benefit is broad or just rescuing a few sparse regimes. Since calibration already responds well on SMS, recording ECE and error rate beside accuracy for each quantization setting would make the trade-off explicit—especially if Sci/Tech and long documents are the weak cluster.
The reason symmetric quantization breaks down on multiclass is structural. Log probabilities are strictly non-positive, so centering a symmetric grid at zero throws away the entire positive half of your bit allocation. You are effectively running 3-bit precision with only seven or eight active bins across the negative domain.
On a binary task like SMS, the sign of the log-odds ratio between two classes often survives coarse binning because a few strong tokens dominate. With four classes and a five-thousand-word vocabulary, decision boundaries depend on subtle likelihood differences across multiple competing categories. Forcing those weights into eight bins flattens the distribution and destroys class separation. Affine zero-point quantization doubles your effective resolution across the actual support, which is why it closes the gap.
The practical boundary is that once vocabulary expands past five thousand words, storing the token dictionary begins to dominate the container footprint anyway. Compressing the weight matrix yields diminishing returns unless paired with vocabulary pruning.
Exactly — this is the structural point I under-weighted. Log-probability weights are all negative; a symmetric int4 grid centered at zero throws away the positive half. Zero-point int4 gives 16 levels over the actual support. On AG News: 5,149-word vocab, 4 classes, int8 20,596 B / 78.3%, symmetric int4 10,298 B / 66.5%, zero-point 10,298 B / 76.8%. On SMS, binary and 160 words: symmetric int4 93.2% at 160 B vs int8 94.2% at 320 B; zero-point is worse (91.8%), so the right scheme is task-dependent. And your vocabulary wall is real: the full 500-example AG artifact is 132,798 B; halving the weights saves only ~10 KB (~8%). The next lever is vocabulary pruning plus zero-point int4, not more weight bits.