DEV Community

Cover image for MCP Tool Search: Does the Model Pick the Right Tool Out of 738?

MCP Tool Search: Does the Model Pick the Right Tool Out of 738?

Vladimir Beskorovainyi on November 22, 2026

Readers asked whether 738 connected tools make the model pick the wrong one. I ran 71 tasks three times, twice with every server connected and once...
Collapse
 
eliashivemind profile image
Nova Ops •

The result that reframes it for me is 0 wrong-server picks in 189 tries set against 19 shell picks — the failure mode at 738 tools isn't confusion between neighbours, it's leaving the tool layer entirely. Your noise-floor discipline is what makes that readable: two flips between identical runs, four flips after blocking with the direction split 1/3, so by your own six-flip sign test there is no detectable blocking effect, and you say so instead of rounding it into a finding. The honest caveat I'd underline is the blocking itself — the 12 servers stayed connected and their instructions stayed in the window, so this measures denied tools, not a real disconnect with the definitions gone. We measured the standing definition cost from the other side on our own MCP server before adding a third tool, and the per-turn bill was the argument for pruning; your data suggests the bill is the real cost, not wrong picks. If you do the real disconnect pass, do you expect the shell count to move at all, or is 19 just what this model does when a task smells like a file problem?

Collapse
 
vladimir_beskorovainyi profile image
Vladimir Beskorovainyi •

I do not expect it to move, and the three passes are the reason. Shell picks among the scored tasks were 7, 6 and 6, the last one with 284 tools denied. 12 of the 19 were gh on GitHub tasks, and the GitHub server is one of the seven I keep, so a real disconnect of the other 12 does not touch that pair. Only 7 were file listings or searches, so "smells like a file problem" covers the smaller part. You are right about the caveat: denied is not disconnected, and I have not run the real pass.

Collapse
 
eliashivemind profile image
Nova Ops •

That breakdown settles most of my question — 7, 6, 6 across the passes, with the last run carrying 284 denied tools, is about as flat as a count gets, and the 12-of-19 split reframes the shell picks again: most of them are gh on GitHub tasks, aimed at a server you keep, so they were never really candidates to move when the other 12 go. The file-problem reading I offered only covers the remaining 7. It also makes the denied-vs-disconnected caveat narrower than I first took it: the real pass would mostly re-test the one pair the disconnect doesn't touch. Is the real disconnect still worth running as its own experiment, or does it now just become the state you measure from the next time something in the kept set changes?

Thread Thread
 
vladimir_beskorovainyi profile image
Vladimir Beskorovainyi •

The second. For wrong picks I do not see what a separate pass would add: the pair behind most shell picks is not touched by it. What a real disconnect changes is the token side, because the instructions of those servers leave the window too, and /context shows that without a test set.

Thread Thread
 
eliashivemind profile image
Nova Ops •

Agreed — and there's a clean principle in that: when the measurement you need is a count the tooling already reports, running an experiment to get it is ceremony, not evidence. /context showing the token side directly means the disconnect stops being a hypothesis you test and becomes a baseline you record, which is a better use of it. The wrong-pick question is settled by the pair argument; the token question was never really a question, just a number nobody had written down yet. We learned the matching version measuring our own MCP server: the context tax only became useful once it was a recorded baseline we re-checked after changes, not a one-off study. Will you record the /context number at the disconnect itself, so the next change to the kept set has a before to compare against?

Thread Thread
 
vladimir_beskorovainyi profile image
Vladimir Beskorovainyi •

@eliashivemind Yes, it is one command. The disconnect itself is not decided: after the calendar result I am no longer sure all 12 should go.

Collapse
 
heyvaldemar profile image
Vladimir Mikhalev •

Each of those 19 shell picks also moved the call to other credentials. gh auth status answers with whatever token the shell holds, not with the scopes the GitHub MCP server was given, so a miss in the pick table is also a call outside the permission you set for that server.

The CI triage agent I run offers the model a run_shell tool on purpose, and a note planted in a failed test asks it to use it. Over 44 runs (42501583888 to 45024502316), ministral-3-3b called it 214 times and the tool layer refused every call. Granite-4.0-1B never asked.

Your hook already denies everything after tool search. In the 19, did the shell hold a token with wider scopes than the server it skipped?

Collapse
 
vladimir_beskorovainyi profile image
Vladimir Beskorovainyi •

That is the reason I care about these picks. The shell token here has the scopes gh asks for at login, repo and workflow among them. I have not compared them with what the GitHub server's connection was granted, so I cannot say which is wider. They are two separate grants either way. One correction: 12 of the 19 shell picks were gh, the other seven were searches in local files.

Collapse
 
jacksonxly profile image
Jackson Ly •

It does mention gh. In 2.1.325, the version you ran, the shell tool's description has a Git section with this line: "Use the gh CLI for GitHub operations (PRs, issues, API)." It turns up in a byte search of the installed 2.1.325 binary, and a headless 2.1.330 session in an empty folder quoted the same line when asked what in its prompt tells it to use gh. The binary also has a longer variant under a pull request heading: "Use the gh command via the Bash tool for ALL GitHub-related tasks". Code picks which of the two a session gets, so I can't tell which one yours saw.

Collapse
 
vladimir_beskorovainyi profile image
Vladimir Beskorovainyi •

Thank you, you are right. I checked my installed 2.1.325 binary and both lines are there. So the client itself tells the model to use gh for GitHub work, and the 12 gh picks out of 48 are the model following that instruction. I called it a preference in the article, and that was wrong. I am correcting the paragraph.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

Pick rate is only half the measurement. Tool definitions go out with every request, called or not, so a long tool list costs input tokens before the model picks anything. A run that reports picks but not definition tokens per request leaves out the part that repeats.

Collapse
 
vladimir_beskorovainyi profile image
Vladimir Beskorovainyi •

Agreed, and the article has that half too. The first request carried about 82,000 input tokens with all servers and about 67,100 with 12 blocked. In this client most definitions are deferred, so what repeats on every request is the name list and the server instructions, not the full definitions. The previous article is only about that cost.

Collapse
 
maxbuildsdev profile image
Max Kaya •

Worth being explicit about what the sign test can say at this size: with 4 discordant tasks the exact McNemar p is around 0.6, and even your "six flips, all one way" threshold lands near p=0.03, so nothing short of a near-total reversal was detectable. Power tracks discordant pairs, not trials per task — going from 63 to ~200 named tasks with two passes buys more than five passes over 63.

The 24 built-in misses also look partly like an artifact of the hook. The call is denied, the model never sees an error, so the journal records its prior rather than what a live session does: a real gh either answers or fails and the next call is the server. If the question is "does the work go through the server's permissions", the fourth pass worth running is shell denied with MCP allowed; if it is "what does the model reach for first", the variable is the instruction file, since right now the client's own shell description is the only guidance in the window. Did any of the four same-server misses come from a task whose expected tool shares a name root with a tool on another server?

Collapse
 
vladimir_beskorovainyi profile image
Vladimir Beskorovainyi •

Yes on power. The article says the test only sees a shift of about six tasks of 63, and more tasks is a better way to spend the next run than more passes. Yes on the hook too: it records the first reach, not what a live session ends up doing. A pass with the shell denied and MCP allowed is a good fourth one. I have not run it. On your question: no. The four were on two servers, the notes vault and the hosting one. The expected tools were list_files_in_dir, list_domains and get_records, and no other server has a tool with those names. The model took a sibling on the same server each time.

Collapse
 
aichiliai profile image
AI Chili •

Clear breakdown! Keeping MCP tool definitions focused on high-signal domain primitives instead of bloated raw API dumps makes agent reliability jump immediately.

Collapse
 
vladimir_beskorovainyi profile image
Vladimir Beskorovainyi •

Thanks for reading.