DEV Community

Cover image for Fine-tuning a 322M Model for Coding Agent Safety: What 3 Runs and 2 Failures Taught Me

Fine-tuning a 322M Model for Coding Agent Safety: What 3 Runs and 2 Failures Taught Me

Changsu Seong on November 02, 2026

laya-guard sits between a coding agent and your shell. Every command Claude Code (or Cursor, Codex, …) wants to run goes through a hook first. Rule...
Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Run 1's 'short means dangerous' lesson is the one that generalizes — a classifier learns whatever shortcut the data offers, and 86% allow validation accuracy hid it completely. The run-2 comment trap is nastier though: 31 of 36 commented commands were allow, so the model learned comments mean routine. Do you think the real fix is more balanced data, or does a guard classifier need features that can't be gamed by surface patterns like length and comments?

Collapse
 
scs0209 profile image
Changsu Seong •

Both, but mostly the second. Comments are text the agent controls, so laya-guard strips them before the model sees a shell command. The regex rules still check the original. Length works the same way: rules run first and can't be talked down, and a block only the model reached becomes a question. Balanced data keeps the model from getting lazy. The pipeline makes sure the shortcuts it still finds can't override the rules.

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Rules-first with the model as a question-raiser rather than a decider is the right hierarchy — the model can still find new shortcuts, it just can't spend them. The part I'd watch is the rules themselves: when the model's questions keep pointing at the same gap, who reviews and updates the regex rules?

Thread Thread
 
scs0209 profile image
Changsu Seong •

Right now, me. Gaps come in through issues and the "Commands laya-guard gets wrong" discussion. Every rule change ships with a regression case and an updated snapshot of what each rule decides, so a fix can't quietly change other verdicts. Nothing turns repeated model questions into rule proposals yet. That's a fair gap to point at.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Honest answer, and the gap you name is the interesting one — if repeated model questions already point at rule gaps, mining them into draft rule proposals feels like the natural next step, with you still approving each one. Do you see that as automation you'd want, or does hand-writing every rule stay a feature, not a bottleneck?

Thread Thread
 
scs0209 profile image
Changsu Seong •

I'd want draft proposals, with me approving each one. A regex is easy to get subtly wrong, so the regression case and the snapshot check stay on me either way. The catch is privacy: the verdict log keeps only a hash and a length, never the command text, so there's nothing to mine yet. It would have to run locally, maybe on top of laya-guard audit, which already reads the agents' own transcripts on your machine.

Collapse
 
bloqarl profile image
Carlos | Zealynx •

gh auth token has a Foundry twin: cast wallet decrypt-keystore deployer prints the deployer's private key, and it has a short alias too, cast wallet dk. It never names a file, it reads ~/.foundry/keystores by account name, so a path rule like credential-read (.ssh, .aws) won't see it.

forge script is the other one. Without --broadcast it's only a simulation, with it the deploy goes on chain and there's no undo. Did any web3 commands show up in your 13,000? If not I'd put both in rules and not leave them to the model.

Collapse
 
scs0209 profile image
Changsu Seong •

Good catch. I just tested both with 0.7.0, and you're right.
cast wallet decrypt-keystore deployer is classified as allow (risk 0.38) even with the model enabled. cast wallet dk triggers a confirmation, but barely (0.46). And forge script scores around 0.5 regardless of whether --broadcast is present, so the model doesn't seem to distinguish between a simulation and an actual deployment.
As for your question, zero. There isn't a single cast, forge, or anvil command in the training data, so the model never had a chance to learn these distinctions.
I agree these are better handled by explicit rules rather than relying on the model. Treating decrypt-keystore and dk as credential reads, and requiring confirmation for forge script ... --broadcast (and probably cast send as well), makes sense.
I'll give this another try based on your suggestions. Thanks for the helpful feedback!

Collapse
 
bloqarl profile image
Carlos | Zealynx •

For the cast send rule, the RPC may not be on the command line at all. cast loads the project's .env by itself and falls back to ETH_RPC_URL when there's no --rpc-url, so a plain cast send from the repo folder can go to mainnet. A rule that reads a missing --rpc-url as local anvil would wave it through.

Thread Thread
 
scs0209 profile image
Changsu Seong •

Thanks, I took your advice and added them as rules. And that’s the reason the cast send rule doesn’t look at the RPC at all. It asks every time, with or without --rpc-url, so a plain cast send that picks up mainnet from .env gets stopped too. It means an extra prompt on anvil, but I’d take that over guessing the chain.

Collapse
 
alexshev profile image
Alex Shev •

The strongest part is treating evaluation failures as evidence about the data-generating process, not merely tuning loss. I’d keep a versioned slice dashboard for routing lane, shell/OS, command length, comment presence, language, and secret-bearing behavior, with regression gates per slice. Aggregate attack recall can stay flat while one newly common shell dialect becomes an unobserved bypass.

Collapse
 
scs0209 profile image
Changsu Seong •

Thanks, and run 2 is pretty much your last sentence. The overall numbers looked fine, but the misses were all in one place. They were Windows commands, which weren’t in the data at all, and commands with a comment in front. I only found them because I went looking by hand. Splitting the results by shell/OS, length and comment presence would have shown that right away instead of me stumbling onto it.

Collapse
 
alexshev profile image
Alex Shev •

That’s exactly the hidden cluster aggregate metrics erase. I’d promote those dimensions into a fixed evaluation matrix and track the per-slice floor, not just the macro score—then a regression in an unseen shell or comment form becomes a release blocker. A small “unknown/other” bucket is useful too: it shows when production needs new coverage rather than just a different threshold.

Thread Thread
 
scs0209 profile image
Changsu Seong •

The unknown bucket is the part I like most. In run 2 the Windows commands would have landed right there, and that was a coverage problem, not a threshold one. No amount of tuning the cutoff would have fixed a shell the model had never seen. Thanks for spelling it out.

Collapse
 
glenallen profile image
Glen Allen •

One of the most valuable takeaways here is that improving safety can easily introduce usability problems if the training data isn't balanced. The shift from being too strict to too lenient shows why evaluating both false positives and false negatives matters. I also appreciate the decision to document the remaining regressions instead of repeatedly tuning against the same evaluation set. That kind of evaluation discipline is essential for building reliable AI-powered developer tools.

Collapse
 
scs0209 profile image
Changsu Seong •

One thing to note is that since the evaluation sets were reviewed between runs, they aren't strictly independent test sets. This is documented in the model card. Instead, I locked down the pass criteria beforehand, logged all misses, and kept the allowed and held counts separate so neither metric could mask the other.

Collapse
 
onelifevibe profile image
OneLifeVibe •

This is a wonderfully honest write-up. The run 1 diagnosis — that the model learned "short means dangerous" because every hand-written block example was short while real allow commands were long — is the kind of failure that only shows up if you actually probe the model on commands it has never seen, and validation accuracy jumping from 0.61 to 0.90 while the real behaviour got worse is a lesson worth repeating. I also appreciated writing the pass criteria down before the first run, and stopping at three runs instead of tuning to the test set. Rules first, model as a question-raiser, and a fresh test set for run 4 is exactly the right discipline.

Collapse
 
scs0209 profile image
Changsu Seong •

Thank you, really glad it was useful. Writing the pass criteria down first ended up mattering more than I expected. Without it, 0.90 would have been very easy to call a win.

Collapse
 
reidmarlow profile image
Reid Marlow •

The length confounding diagnosis in run 1 matches what happens whenever synthetic failure cases get written by hand. The other structural wall with classifying agent commands at the shell boundary is script indirection. When an agent hits restrictions on direct shell syntax, it often falls back to Python or Node one-liners to touch files. A 322M model classifies python3 -c as a standard build step, while the actual filesystem mutations sit inside string arguments or dynamic imports. Pairing the classifier with container isolation or dropping write access outside the workspace git tree catches the escaped mutations that semantic parsing skips.

Collapse
 
scs0209 profile image
Changsu Seong •

I haven’t really tested it enough to say whether it holds up.
My eval sets have no attack commands longer than 300 characters, and only six between 80 and 300 (four of them are in run 3). The held-out split has only three long, risky scripts, and run 3 catches two of them. It also still flags about half of the long, harmless scripts (24 out of 46 that should be allowed). So length is probably still playing a role.
I think your approach is better than what I did. The “long script with one bad line” examples in run 2 were hand-written. Splicing a destructive step into real, long, allowed scripts keeps everything else the same, so the main difference is just the bad step itself.
I’ll use this approach for the next training set and create a separate test set that I won’t touch while tuning.

Collapse
 
advikdivekar profile image
Advik Divekar •

Recently I came across this saying that FT actually makes the model perform bad. Nothing towards you but can you please share me your thought on process during the Fine-Tuning stage

Collapse
 
scs0209 profile image
Changsu Seong •

No offense taken, it actually did make it worse at first.

The task is narrow: one command in, allow/ask/block out. So I wasn't worried about the model forgetting general knowledge, I was worried about it learning the wrong shortcut. And it did. In run 1 most of my data was "allow", and every risky example I'd written was short while real commands were long scripts. Validation looked great, but on held-out sets it got worse and started asking about git status. It had learned "short = dangerous".

Run 2 fixed that and overshot, it let a Windows drive wipe through because the data had no Windows commands. Run 3 balanced the labels and filled those gaps. Overall it beats the base model on both everyday commands and attacks, but not everywhere: chmod +x on a build script now gets a question it didn't get before, and it still asks about about half of the long harmless scripts. So it's better, not fixed.

Run 1 looked really good on paper. If I hadn't tried it on commands it had never seen, I'd probably have shipped it. That's also why I don't let the model have the last word on the really bad stuff. Something like rm -rf ~ gets caught by the rules before the model even sees it.

Collapse
 
advikdivekar profile image
Advik Divekar •

Thank you so much @scs0209!!
Now that you mentioned in Run 2 and 3 it got better, what exact changes were done in Run 2 in order to avoid the Run 1 happenings and also for Run 3.
Did Run 3 rectified the errors of Run 2 or both Run 2 and Run 3 gradually fixed the Run 1s issues.

Thread Thread
 
scs0209 profile image
Changsu Seong •

Both, in sequence. Run 2 fixed Run 1's false alarms and overshot, and Run 3 kept that fix and corrected the overshoot.

Run 2 added 206 examples aimed at Run 1's "short means dangerous": 98 short, ordinary commands, long scripts with one harmful line in the middle, and 69 Korean and English requests. Everyday commands jumped to 49/50, but allow grew to 84% of the rows, and rmdir /s /q d:\, a harmful command behind a comment, and gh auth token got through.

Run 3 kept Run 2's data and changed three things:

  1. Dropped the 242 safe-lane commands, which never reach the model in use.
  2. Counted review and block rows twice in training, which took allow from 84% to 61%.
  3. Added 100 examples for Run 2's gaps: Windows commands, secrets printed or sent, harm behind a comment, tools told to run something else, and read-only Korean requests.

Run 2's misses were all held again, and Run 1's gain on attacks came back (39/43) without its false alarms (44/50 everyday).

Collapse
 
crisstiank profile image
Cristian K •

The length confound is the part I found most useful: real commands labelled allow were long, hand-written risky ones were short, so the model learned "short means dangerous". A practical addition is to also pull risky variants from the real command distribution, for example by taking long allowed scripts and inserting a destructive step, so length stops being a free signal. Did run 3 hold up on those long mixed commands?

Collapse
 
scs0209 profile image
Changsu Seong •

That makes sense. The difference between blocking a straightforward command and catching the same operation hidden behind encoding or indirection is pretty significant.
I hadn't considered enforcing those restrictions at the kernel level through the agent's own sandbox. That seems like a much stronger approach than relying on semantic classification alone, especially when the model or rules can't reliably inspect what a script actually does.
I also think the stricter mode that prevents writes outside the git tree is worth exploring. I'll take another look at this approach and see how I can incorporate it.
Thanks for taking the time to explain how this works. Really helpful!