Behavioral vaccination: a 400-row mixed prime protects a behavior

This is a cross-disciplinary experiment, not a container. Can a small "vaccine" adapter stop
a later topic adapter from erasing a sentiment behavior?

The first run timed out while evaluating one prompt at a time. The second run batches
candidate sequences and reuses the trained prime. The full 400-row test now completes.

Reproduce:

python research/vaccination.py        # trains the prime (63 s on the 6 GB laptop)
python research/vaccination_eval.py   # batched full evaluation -> vaccination.json

Setup

• Host: 4-bit Qwen3B.
• Prime ("vaccine"): LoRA r=16, one epoch on 300 SST-2 + 100 AG News rows.
• Prime training: 63.2 s for 398 encoded rows; adapter 29.6 MB.
• Pathogen: the existing full AG News adapter.
• Merges: 0.50 and 0.75 prime weight.
• Evaluation: 400 SST-2 + 400 AG News rows, batched label-token logit scoring.

Results

The effect is real on this test

McNemar against the AG adapter alone:

• SST-2: +200 vs +104 discordant, p < 0.001.
• AG News: +111 vs +9 discordant, p < 0.001.

Against the base:

• SST-2: +53 vs +4, p < 0.001.

The vaccine merge is worse than the dedicated SST-2 merge on sentiment (74.0 vs 94.5,
p < 0.001), but better on AG News (71.2 vs 67.0). A 4-bit pack loses one SST-2 row
(73.5 vs 74.0, p = 1.0) and gains 0.6 pp on AG; the behavior survives quantization.

Composition matters: 25 / 50 / 100 pathogen rows

I trained primes with 25, 50 and 100 AG News rows (plus 300 SST-2), then repeated the
0.75 merge:

This is not a monotonic dose. The 50-row prime is sentiment-heavy (86.5% alone) and
gives the best sentiment under the pathogen (76.0%). The 100-row prime is topic-heavy
(72.0% AG alone) and gives the best AG under the pathogen (71.2%). The 25-row prime is too
weak: it rescues sentiment against the AG adapter (p=0.006) but not against the base
(64.0 vs 61.8, p=0.33).

The composition of the prime, not just its size, sets the merge compromise.

Held-out pathogen: logit-healthy, generation-pathogenic

To test a true prime–boost, I trained AG News adapters on disjoint rows the 50-row vaccine
never saw: 300 rows (75/class) and 1,000 rows (250/class).

Under logit scoring, the held-out adapters look healthy: good AG accuracy and preserved
sentiment. The 1k adapter is a better topic adapter than the original 6k one.

Generation tells a different story. On a balanced 400-row SST-2 check (200 positive,
200 negative), batched greedy generation:

The 1k adapter is benign under logit ranking (84.5%) and pathogenic under generation
(55.8%). It predicts positive on 375/400 rows. The vaccine merges repair most of that:
83.5% and 85.5%. The dedicated SST-2 adapter is still higher (93.75% generation on 400 rows
in the earlier SST-2 run), but the vaccine is a broad compromise.

The original 6k adapter is 0/400 on generation (all unknown) and 50% on logit. It is not
simply "failed": its AG generation accuracy is 62.25% vs base 2.5%, while its logit AG score
is 45.8% vs base 59.8%. It is a working generative topic adapter with bad label-token
ranking — and it erases sentiment generation completely.

The 1k pathogen is the true held-out test: the vaccine never saw its 300–1k rows, yet
the generation score goes 55.8% → 85.5% under the 0.75 merge. That is a held-out,
generation-level vaccination rescue on 400 rows, not 8.

The lesson is metric-dependent interference. A logit probe can say "healthy" while
generation says "broken." The vaccine rescues the generation behavior in both merges, now
on 100 rows, not just 8.

The strong-AG vaccination result still stands for that adapter, but the vaccine saw 50 of
its training rows; it is not evidence of pathogen-agnostic immunity.

What it looks like

• The mixed prime is a broad-spectrum adapter: 82.2% SST-2 and 72.0% AG News alone.
• Adding the AG pathogen at 0.75 rescues sentiment from 50.0% to 74.0% and lifts AG News from 45.8% to 71.2%. Both effects are significant on 400 rows.
• The prime composition matters: 50 AG rows is the sentiment sweet spot (76.0%), 100 AG rows is the topic sweet spot (71.2%), and 25 AG rows is too weak (64.0%).
• This is a multivalent vaccine, not a true prime–boost: the prime already saw 100 AG News rows. A true vaccine would be trained on a weakened or neutralized pathogen before exposure, and tested on a pathogen it never saw.
• The dedicated sentiment adapter is still the best sentiment carrier; the vaccine is a compromise that carries both behaviors.

Caveats

• One seed, one host, one prime size (400 rows), one epoch; no cross-validation.
• Main table is logit scoring, 400 rows per task; the generation check is 400 balanced rows.
• Logit scoring and generation disagree strongly on the held-out 1k adapter (84.5% logit vs 54.0% generation); both are reported.
• The prime contains the pathogen's task data, so this is not evidence of pathogen-agnostic immunity.
• q4/q8 were packed from the float merge; they are not independently trained.
• Battery note: the first run dropped the laptop from 72% to 23%; the second run was done with the charger connected.

Next experiments

• True prime–boost: train the prime on a weakened AG subset (or a sentiment-preserving topic set), then expose the full AG adapter. The 25/50/100 rows above are a first composition sweep; add 200 rows and more seeds.
• The 1k held-out generation rescue is now 400 rows; add more seeds and a second host.
• Re-run the original 6k recipe and inspect its loss/optimizer trajectory; is the erasure a training-scale effect, or a metric effect?
• Pathogen-agnostic immunity: prime on task A and test against adapters B/C.
• More seeds; the current numbers are one run.
• Compare vaccine sizes (100/400/1,000 rows) and merge ratios.
• Test whether the vaccine survives a second pathogen (sequential exposure).

The result is not "we invented immunity." It is a controlled demonstration that a small
mixed adapter can shift the merge compromise toward keeping both behaviors, and that the
effect survives 4-bit packing.