DEV Community

Your coding agent has opinions. Log the calls it made for you

Human On Mute 401 on November 12, 2026

Code review is good at one thing: finding mistakes. A failing test, an off-by-one, a missing null check. Something is wrong, and a reviewer can poi...
Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The 86% match rate against a coin flip is the damning stat — the agent's lean predicted the final recommendation better than the analyst's own expertise did. The judgement-call log is a good countermeasure, but the 'argument came from' column depends on the analyst noticing the influence in the first place. Do you think the log actually works when 77% of people repeat the agent's argument as their own reasoning — or does it just document the deferral after the fact?

Collapse
 
humanonmute401 profile image
Human On Mute 401 •

You're right. The 'argument came from' column is the weakest part, for the reason the study found. If 77% repeat the agent's case as their own and only 15% credit it, a source label added after the fact will often miss the influence.

I put more weight on two other habits. 'Result under the other option' puts the rejected path in the record, even if you never noticed who argued for the chosen one. The self-test, a fresh session that has to argue the option you rejected, is meant to catch thin spots before merge.

The log will sometimes only document deferral after the fact. A reviewer can still see a thin 'other option' row even when the author cannot.

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The reviewer angle is what saves the log for me — even if the author can't see their own deferral, a thin 'other option' row is visible to someone else. The self-test is the stronger habit of the two, but it hinges on one detail: does the fresh session argue the rejected option blind, or does it see the agent's case first? If it sees the case, it's just re-reading the influence rather than testing it.

Thread Thread
 
humanonmute401 profile image
Human On Mute 401 •

Blind. That's why it's a fresh session: no history, none of the original agent's case pasted in, just a neutral framing of the decision and a request for the strongest case for the option you rejected. If it reads the first argument, you're testing whether a model agrees with itself. Even blind, it's a rough check. The new session has its own priors, so it shows you where your view is thin rather than proving the call was right.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Blind is the only way the test means anything — the moment the new session sees the original case, you're just measuring agreement with yourself. The priors caveat is fair, though: a single blind run shows where your view is thin, but does one run tell you enough, or do you repeat it to separate a genuinely thin argument from the new session's own lean?

Thread Thread
 
humanonmute401 profile image
Human On Mute 401 •

One run is enough for what the self-test is for: a prompt to look again before you merge. If the case for the rejected option reads as weak, move on. If it lands, that's when a second run earns its place. Reword the framing, swap the order of the options, or try a different model, and see whether the argument survives. If it holds up across all three, your view probably is thin. If it only shows up under one framing, that's more likely the session's lean. Either way, flag the row for the reviewer rather than settling it alone.

Collapse
 
deanlee profile image
Dean Lee •

The unseeded baseline of 112 identical choice sets across 123 runs of Claude Opus 4.8 exposes a correlated-prior risk alongside individual deferral. Even without an explicit nudge, teams running the same default model converge on identical retry locations, caching boundaries, and timeout thresholds. When a reviewer uses that same model to audit the pull request, two matching approvals reflect one shared model prior.

Adding the 'Result under the other option' column addresses that blind spot by putting the cost and failure profile of the rejected path directly in the review record. Once the tail risk of duplicate client writes or database read spikes is written down next to the chosen approach, the team can price the tradeoff explicitly.

Collapse
 
humanonmute401 profile image
Human On Mute 401 •

That's the risk I was trying to get at with the Independent check column. The study measured deferral after a deliberate lean, but the unseeded runs already show a strong default: the same five choices in 112 of 123 cases. If the author and the reviewer are both prompting the same model, agreement is not two views.

Writing down the result under the other option is the practical fix, as you say. Once the rejected path's cost or failure mode is on the page, the team can argue about the tradeoff rather than nod through a shared default. The log does not remove the prior. It just makes it visible enough to challenge.