A reader named izgorodin left a comment on my piece about the 200-tool menu, and it changed the question I thought I was answering. He had run the numbers himself: forty turns of a triage helper whose only tool returns five GitHub issues, and by turn twenty the tool results were 99.1 percent of everything sent, about 326,000 tokens, thirteen times the cost of the menu itself. Keeping only the last six turns flattened the call at around 99,000. Trimming each result to the five fields the helper actually reads kept the whole history and ended turn forty at 36,934. Then he wrote the sentence that matters: his replay called no model and used a fixed answer, so it measured how big the window gets, not whether the helper still finds what it needs.
That is the question I spent this week answering. Same twelve triage jobs, run twice: once with the full issue text coming back, once with each result cut to the five fields (the number, the title, the labels, the comment count, and the first 300 characters of the body). A real model did the work, a small one I run locally, and every job was scored on whether it found the right issue and the right label. I expected the trim to cost something. I was wrong about the direction.
My claim before the numbers: trimming the results would save about half the tokens per turn and cost a few solved jobs, because sooner or later the helper would need something the trim cut. The first half was right. The second half was backwards.
Problem 1: the trim did not cost a single solved job
Across both runs, the full results solved 9 of 24 jobs. The trimmed results solved 17 of 24. That is 37.5 percent against 70.8, and the gap is not a rounding error in one direction: the trim won 9 to 4 in the first run and 8 to 5 in the second. Look at the twelve jobs one by one and the trim never loses a job the full text wins. The two arms agree on seven jobs (both solve four, both fail three), and on the other five the trim wins or ties every time.
Three jobs beat the model in both arms, and they are worth naming because they set the limit. On the Linux dependency guide, the Windows 11 question, and the misspelled error dialog, the model answered the crash-on-launch issue every single time, in both arms, in both runs. Those are small-model limits, not trim damage. The trim cannot be blamed for what the full text also fails, and the fact that the failures are identical in both arms is the cleanest evidence in the whole run that the trim is not hiding anything load-bearing.
Problem 2: the full text has a favourite issue, and it is always the crash
The full arm's failures have a shape, and once you see it you cannot unsee it. On five of its failed jobs, the model answered issue 412, the crash on launch, with the label "bug" — whatever the job actually asked. The job about garbled dates in a CSV export? 412, bug. The job about the install guide missing a Linux step? 412, bug. The job about a typo in an error dialog? 412, bug. It even went back and read 412 a second time before answering it, as if checking its work.
The trimmed arm does the same second look just as often, about once per job in both arms, but it answers the right issue afterwards. The difference is not that the trim hides the crash report. It is that five short results sit side by side and all five stay visible, while five long ones let the most dramatic report swallow the rest. A small model reading a long list does not weigh five issues. It remembers the loudest one. Cutting the noise did not just save tokens. It changed what the model could pay attention to, and attention was the scarce thing, not information.
Problem 3: the full text can eat the whole window just by looping
The ugliest full-arm result is not a wrong answer. It is no answer at all. On two jobs the model listed the issues, listed them again, and listed them again, six turns of the same call, and burned 10,063 tokens on one job before the turn cap stopped it. The first time I ran this, that loop pushed the prompt past the model's whole window and the run died outright. I added a rule that a repeated call comes back as a one-line note instead of the full dump again, the same rule in both arms, and the run survived, but the loop still cost the job.
The trimmed arm never needed a fourth turn on any job. Every job ended in three: list, one second look, answer. The full arm averaged close to four turns and needed six on its worst jobs, and each of those extra turns carried another thousand-plus tokens of history with it. The token bill per turn tells the same story from the other side. Turn one is identical in both arms, about 179 tokens. Turn two is 1,749 with full results and 759 trimmed. By the third turn the full arm is sending about 1,900 tokens a turn and the trimmed arm about 1,080. The average across the run is 1,421 tokens per turn against 673, and 5,475 per job against 2,020. The trim cuts the bill by more than half per turn and almost two thirds per job, and the jobs it saves are the ones where the full text was already losing.
Problem 4: the second look is not the price of the trim
izgorodin asked for a miss signal alongside the solved rate, and he was right to: a trim that quietly sends the helper back for the full text every time has not saved anything, it has just moved the cost. So I counted every time the helper asked for one issue in full. Full results: 0.88 second looks per job. Trimmed results: 0.92. That is the same number, inside the wobble between runs, and it is the number that surprised me most. The trim did not drive the helper back for more. The helper goes back about once per job in both arms, usually to look once more at the issue it is about to answer, and the full text costs the same second look for the same reason.
That kills the natural objection in both directions. The trim is not free because the helper never needs the cut material: it goes back just as often. And it is not expensive because of the second looks, because the second looks cost the same either way. The saving is in the list itself, paid once per turn, and the second look is a fixed cost of doing the job carefully, not a tax on trimming.
Problem 5: the honest limit is the size of the model
All of this is one small model on twelve jobs. A bigger model would solve more jobs in both arms (the three jobs that beat this one in both arms would probably fall), and the gap between the arms would likely shrink. The absolute rates, 38 percent and 71 percent, belong to this model and these twelve jobs, and anyone quoting them as general truths about agents is over-reading a small run. Twelve jobs is also small enough that the run-to-run wobble is a whole job per arm. I ran it twice for that reason, and the direction held in both runs, but the direction is the finding, not the percentages.
The other limit is the jobs themselves. My issue bodies run a hundred-odd words, shorter than real reports. Real bodies are longer, messier, and more repetitive, which would widen the token gap, not narrow it. But it would also give the trim more to cut, and I have not measured that. The five fields are also izgorodin's five, chosen for a triage prompt that cites numbers and suggests one existing label. A different prompt reads different fields, and its trim would cut somewhere else.
So the rule I am keeping is narrower than the one I went in with. Trim the result to the fields the prompt actually reads, and measure the solved rate before you argue about it, because the cost you expect (the helper going back for what you cut) may not exist, and the benefit you did not expect (the helper deciding better with less noise) may be the whole story. The menu was never the bill. The results were. And the results, it turns out, were also the distraction.
What do you trim from your tool results, and did you check whether the solved rate moved when you did? I would like to hear the number, either way, and what the helper went back for when it went back.
P.S. Everything I write stays free — that part doesn't change. But if this one was worth something to you, you can buy me a coffee ☕ (one-time or monthly, both work). It covers the API credits and hosting behind these experiments — and honestly, it's also the nudge that gets the next one started on the evenings I'd otherwise skip it. — Salam


Top comments (15)
The "favourite issue" failure is the part I'll remember from this. It reads like salience rather than missing information: five long results, and the most dramatic one wins regardless of the question. That also explains why the failures were identical in both arms on the three hard jobs: the trim removed noise, not signal.
Did you try a third arm: full text kept, but the five-field summary of every result placed first? It would separate the two effects you're seeing, whether it's the length of the results or their position in the window that pulls the model toward issue 412.
The salience angle is the part that stuck with me too — five long results and the dramatic one wins no matter what was asked, that's not missing info. Haven't run the summary-first arm, and it's the clean split between length and position. Going to run that one, same tasks, three arms. Which would you bet on — the length of the results, or where they sit in the window?
My bet is position. In my agents the first long result tends to anchor everything after it, and the later ones get skimmed. So I'd expect summary-first to beat full text even at the same token count. If length turns out to matter more, that's the more interesting result. Please post the numbers when you run it.
Position's my bet too, honestly — the first long result sets the frame and everything after gets read through it. Same token count, summary-first vs full text, that's the clean split. I'll post the numbers when the run's done — if position wins, would that change how you order results in your own agents, or is it just something you live with?
Honestly, yes, it would 🙂 If position wins, I'd lead with a one-line summary of each result and put the long bodies after, so no single result gets to set the frame. If it's length, I'd just cap each result and move on. Really curious what your three arms show, ping me when the numbers are in!
Numbers are in — trimmed took 17 of 24 against 9 full, and the second-look rate barely moved either way. Your summary-first version is the arm that actually tests the position bet, so that's next. Still think position, or did the 17 change your mind?
This matches what we've seen with web extraction and search tools too: returning raw markdown/HTML or full JSON payloads makes the model anchor on whatever happens to be longest or loudest in the context window, rather than what actually answers the query. Trimming to a tight schema (or truncating bodies + returning only load-bearing fields) improves both accuracy and token cost at the same time—especially on multi-turn loops where old tool outputs compound fast.
Same shape, different tool — that's reassuring, actually. The multi-turn compounding is the bit that surprised me most, old outputs just sitting there getting heavier every turn. Did you measure the accuracy side too, or was it mostly the token bill that pushed you to trim?
Token bill was what forced the change first — once a loop hit 15–20 turns with full web/search payloads, input tokens were dominating the cost and pushing smaller models near their context ceiling. The accuracy improvement showed up right after as a side effect: fewer "lost in the middle" misses and far fewer retry loops where the agent re-called the same search because the previous result was buried under boilerplate. Haven't run a formal two-arm A/B like yours with scored pass rates, though — mostly observed it from production traces dropping in turn count once we capped returned fields.
The paired per-job outcomes are more useful than the headline 17/24 at this sample size. A useful limit-case job would put the decisive clue after the first 300 body characters, where the five-field list really does discard information. Then you can see whether the second look repairs the miss, how many tokens it costs, and whether the full arm wins that specific case. Would you include one deliberately late-clue issue alongside the summary-first arm?
That's the job the test needs — one where the answer sits past the 300 characters and the trim actually has to earn it. Going to put a late-clue issue in the summary-first arm and see if the second look repairs the miss or the full arm takes that one. Where would you hide the clue — buried in a long repro, or right at the tail?
The result that surprised me most is not the token saving but the 0.88 versus 0.92 second-look rate — the trim did not make the agent lazier about verifying, which was the objection I would have raised first. One detail I keep coming back to is that the five kept fields still include the first 300 characters of the body, so issue 412's "crash on launch" signal survives the trim in miniature. That makes me wonder whether the remaining salience effect is carried by that excerpt rather than by length itself, and whether a title-plus-labels-only arm would change the 17 of 24. If the excerpt is doing the work, the rule might be less "trim everything" and more "keep the shortest field that still lets the model rank issues". Did the trimmed arm's five remaining failures cluster on jobs where the decisive detail sat beyond those 300 characters?
That's the sharpest read on it so far, but the failures don't actually back the excerpt theory. The three jobs the trimmed arm fails are the same three the full arm fails. The model just kept answering 412 no matter what it got, so the 300-char cutoff isn't what broke those. The title+labels-only arm is the one that settles your question though, and it's next on my list. If the excerpt is doing the work, where would you bet it lands, closer to 17 or back down near 9?
Your run answers the half my replay could not: mine called no model, so it measured how the window grows and nothing about the answers. One detail in your setup makes the token gap a lower bound. The rule that turns a repeated call into a one-line note was on in both arms, and before you added it the full arm ran past the window and died. So 1,421 against 673 tokens per turn is the full arm with its worst habit already trimmed. If you publish the per-job table for both runs together with the title-plus-labels arm, readers can see on which jobs the arms disagree, and with twelve jobs that is where the result lives.
Fair catch — that 1,421 against 673 is the polite version, the full arm with its worst habit already fixed. The per-job table goes up with the title-plus-labels arm, that's the run that makes it worth reading. My bet is the arms disagree on the jobs where the answer hides past the 300 characters — where's yours?