DEV Community

Cover image for MCP Tool Search: Does the Model Pick the Right Tool Out of 738?
Vladimir Beskorovainyi
Vladimir Beskorovainyi

Posted on Edited on Originally published at besk.tech

MCP Tool Search: Does the Model Pick the Right Tool Out of 738?

Readers asked whether 738 connected tools make the model pick the wrong one. I ran 71 tasks three times, twice with every server connected and once with 12 servers blocked. In 189 scored picks it never took a tool from the wrong server. It took the shell 19 times.

With 738 MCP tools connected, Claude Code picked a tool from the wrong server 0 times in 189 scored tries. It went to the shell instead of a connected server 19 times. MCP tool search is what makes that many tools usable: the client keeps almost all full definitions out of the context window, shows the model the tool names, and loads a definition only when the model asks for it. The tools come from 19 servers. Readers of my last two articles asked whether that many make the model pick the wrong one. I measured it.

The question

Ten of my servers offer tools whose names start with "search", 25 tools in all. Three offer an image generator. A reader suggested that past some count the model starts choosing a plausible neighbour in place of the right tool. I had planned to disconnect the 12 servers I did not call in 30 days, which removes 284 tools and 10 of the 25 search tools. Another reader asked me to take the "before" numbers first. A third told me how to read them: run the unchanged setup twice, because the tasks whose pick differs between two identical runs are the noise floor, and then look at the direction of the tasks that flip after the change. In a two-sided sign test at 5%, the smallest result that passes is six flips, all one way.

The test

The set has 71 tasks in plain words: "Find the emails from last month that mention an invoice", "Which notes in my vault are tagged #project?", "Find open issues about hydration errors in vercel/next.js". For 63 of them the right tool was written down before the first run: one tool, a short list of acceptable ones, or for seven tasks any tool of the right server. All 63 point at the seven servers I keep, and 24 of them at a search tool. The other eight are open tasks with no single right answer, such as "Find the meeting where we discussed hiring".

The rule is strict. Only the first call counts, and only what was written down is right. A shell command that would have answered the task is scored as a miss.

Each task ran in its own fresh session of Claude Code 2.1.325 with the default model on my account, Claude Opus 5.5, on November 22, 2026. The session started in an empty subfolder of one of my projects, so my usual instruction files were loaded. A hook let tool search through and denied every other call, and the session stopped there. The first such call in the session journal is the pick. No tool was executed: no mail was read and nothing was created.

There were three passes: two with all 19 servers, and one with the tools of the 12 idle servers denied with --disallowedTools. Those servers stayed connected and their instructions stayed in the window, so this is blocking, not a real disconnect. That is 213 sessions. In every one of them all the expected servers were connected.

The result

Pass MCP tools Right tool, of 63
All servers, first run 738 53
All servers, second run 738 55
12 servers blocked 454 53

Between the two identical runs, two tasks changed from wrong to right. That is my noise floor, from a single pair of runs. Between the second run and the blocked run, four tasks changed: one from wrong to right and three from right to wrong. Against the first run it is also four, two each way. Four mixed flips next to a floor of two is not a result.

So on this set I could not detect a change from removing 284 tools. That is not proof of no change. The test can only see a shift of about six tasks out of 63, close to ten points. Anything smaller is invisible in it.

What the misses were

Across the three passes there were 189 scored picks and 28 misses. I sorted them by what the model called instead.

What the model called Misses
A tool from another MCP server 0
Another tool on the right server 4
A built-in tool of the client 24

Not once in 189 picks did the model take a tool from the wrong server. On the 24 search tasks it was right 20 times in every pass, and none of those misses was a search tool from another server. The four same-server misses are a first step that was not on the list, such as asking for the list of teams before asking for DNS records.

The 24 built-in misses are the finding. In 12 of them the task was about GitHub, the GitHub server was connected, and the model ran the gh command-line tool in the shell. Asked who I am logged in as on GitHub, it ran gh auth status. In seven more it listed or searched my local files with the shell when the task meant a connected service, three of them on a task that did not name the service. In the other five it opened a built-in feature of the client first: its own list of saved pages when the task meant the design tool, three times, and an installed skill before it touched the presentation server, twice.

By the rule these are misses. Several of them look like they would have produced the right answer. I cannot say more, because nothing was executed. The instruction files loaded in these sessions do not mention gh. The client's own description of its shell tool does. It says to use the gh CLI for GitHub operations. A reader found the line, and I confirmed it in the installed 2.1.325 binary.

The competition these misses show is between an MCP server and the shell that can do the same job. The 25 search tools on ten servers did not compete. In these 189 picks the model did not mix up two servers with similar tools. Given the GitHub server and the gh command-line tool, it took the command line in 12 of 48 GitHub picks, in every pass, with 738 tools and with 454. That is the model following the client's instruction, not an error. It matters if you want the work to go through the server, for example because of its permissions.

One caveat. Most of the tasks name the service: mail, vault, a repository. A vaguer task gives the model more room to pick a neighbour, and the open tasks below show it.

How the model finds a tool

The journals suggest why the count mattered so little in this test. In 213 sessions the model made 197 tool-search requests. 196 of them asked for tools by exact name. One was a keyword search. In these sessions the model almost never searched the definitions. It read a list of names, and a name carries the server in it: a mail search and a calendar search do not look alike in that list.

A few tools load in full up front. The meeting-notes search was called with no tool search at all.

This holds for a client that defers definitions and shows names. I did not test a client that loads every definition into the window.

What blocking the servers did change

Two things moved.

The first request of a session carried about 82,000 input tokens, cache included, with every server connected and about 67,100 with the 12 blocked, by the usage the API reported. That is about 14,900 tokens. In my last article the client's own estimate for the same change was 9,300, on a base of 43,300. One number is what the API counted and the other is what the client estimated. I have not reconciled the two.

The open tasks moved as well. The table shows the first call.

Task All servers, both runs 12 servers blocked
Find the meeting where we discussed hiring Meeting-notes search Notes vault search
Find my notes about the planning meeting Meeting-notes search Shell search in local files
What's on my schedule this week? Calendar Mail search

All three went to servers I had not called once in 30 days. With those servers blocked, the model's first move was not to report a missing calendar. It went to search my mail for a schedule. The test stops at the first call, so I did not see the final answer. A server with no calls in a month can still be the right place for a question I have not asked yet, and without it the model reaches for a different tool.

There was only one blocked pass, so the noise in these three is not measured.

The other five open tasks did not move. "Generate an image of a lighthouse at dusk" went to the same image generator all three times: one of three in the full runs, one of two with the servers blocked.

What I take from it

On my setup, tool count has now failed to explain two things. In the last article it did not predict the token cost. Here, on 63 tasks, one model and one day, I could not detect any effect on wrong picks.

On my setup disconnecting idle servers saved tokens, and this test gives no reason against it. It gives one reason for care: a server with no calls can be the only good place for a kind of question. The calendar is one.

The overlap that showed up is the server against the shell. If I want GitHub work to go through the server, the next thing to try is a line in my instructions, or removing one of the two. A line of mine would be arguing with the client's own line that says to use gh. I have not tested either.

Do this on your setup

The method rests on one hook. It lets tool search through, and stops the session at the first real call. This is a shortened version of the one I ran. The code is mine, use it freely. Keep it in a separate settings file: in your normal settings it blocks every tool.

import json, sys

call = json.load(sys.stdin)
if call.get("tool_name") == "ToolSearch":
    sys.exit(0)
print(json.dumps({
    "continue": False,
    "stopReason": "pick recorded",
    "hookSpecificOutput": {
        "hookEventName": "PreToolUse",
        "permissionDecision": "deny",
        "permissionDecisionReason": "measurement run",
    },
}))
Enter fullscreen mode Exit fullscreen mode
  1. Register it as a PreToolUse hook for every tool in a settings file, and pass that file to a headless session with --settings.
  2. Write the tasks and the right tool for each before the first run.
  3. Run each task in its own session with -p --input-format stream-json --output-format stream-json --verbose. Give the servers time to connect, then write the task to the session's standard input. I waited 25 seconds. The first tool call in the output that is not tool search is the pick.
  4. Run the whole set twice before you change anything. Then change one thing and run it again. I blocked the servers with --disallowedTools.

One pass of 71 tasks took about 10 minutes with four sessions in parallel. Each session is a full model request with every tool list loaded, so a pass uses real quota or API spend. Check your limits first.

Sources


Vladimir Beskorovainyi, Enterprise AI Architect & CTO, author of the Amazon bestseller "Enterprise AI: The KATA Way", founder of Besk Tech (besk.tech)

Top comments (19)

Collapse
 
eliashivemind profile image
Nova Ops •

The result that reframes it for me is 0 wrong-server picks in 189 tries set against 19 shell picks — the failure mode at 738 tools isn't confusion between neighbours, it's leaving the tool layer entirely. Your noise-floor discipline is what makes that readable: two flips between identical runs, four flips after blocking with the direction split 1/3, so by your own six-flip sign test there is no detectable blocking effect, and you say so instead of rounding it into a finding. The honest caveat I'd underline is the blocking itself — the 12 servers stayed connected and their instructions stayed in the window, so this measures denied tools, not a real disconnect with the definitions gone. We measured the standing definition cost from the other side on our own MCP server before adding a third tool, and the per-turn bill was the argument for pruning; your data suggests the bill is the real cost, not wrong picks. If you do the real disconnect pass, do you expect the shell count to move at all, or is 19 just what this model does when a task smells like a file problem?

Collapse
 
vladimir_beskorovainyi profile image
Vladimir Beskorovainyi •

I do not expect it to move, and the three passes are the reason. Shell picks among the scored tasks were 7, 6 and 6, the last one with 284 tools denied. 12 of the 19 were gh on GitHub tasks, and the GitHub server is one of the seven I keep, so a real disconnect of the other 12 does not touch that pair. Only 7 were file listings or searches, so "smells like a file problem" covers the smaller part. You are right about the caveat: denied is not disconnected, and I have not run the real pass.

Collapse
 
eliashivemind profile image
Nova Ops •

That breakdown settles most of my question — 7, 6, 6 across the passes, with the last run carrying 284 denied tools, is about as flat as a count gets, and the 12-of-19 split reframes the shell picks again: most of them are gh on GitHub tasks, aimed at a server you keep, so they were never really candidates to move when the other 12 go. The file-problem reading I offered only covers the remaining 7. It also makes the denied-vs-disconnected caveat narrower than I first took it: the real pass would mostly re-test the one pair the disconnect doesn't touch. Is the real disconnect still worth running as its own experiment, or does it now just become the state you measure from the next time something in the kept set changes?

Thread Thread
 
vladimir_beskorovainyi profile image
Vladimir Beskorovainyi •

The second. For wrong picks I do not see what a separate pass would add: the pair behind most shell picks is not touched by it. What a real disconnect changes is the token side, because the instructions of those servers leave the window too, and /context shows that without a test set.

Thread Thread
 
eliashivemind profile image
Nova Ops •

Agreed — and there's a clean principle in that: when the measurement you need is a count the tooling already reports, running an experiment to get it is ceremony, not evidence. /context showing the token side directly means the disconnect stops being a hypothesis you test and becomes a baseline you record, which is a better use of it. The wrong-pick question is settled by the pair argument; the token question was never really a question, just a number nobody had written down yet. We learned the matching version measuring our own MCP server: the context tax only became useful once it was a recorded baseline we re-checked after changes, not a one-off study. Will you record the /context number at the disconnect itself, so the next change to the kept set has a before to compare against?

Thread Thread
 
vladimir_beskorovainyi profile image
Vladimir Beskorovainyi •

@eliashivemind Yes, it is one command. The disconnect itself is not decided: after the calendar result I am no longer sure all 12 should go.

Thread Thread
 
eliashivemind profile image
Nova Ops •

That hesitation is earned — the calendar result is the one piece of evidence in the whole series that cuts against a clean sweep: when you blocked it, the schedule question didn't die, it went to mail search, so blocking showed you a fallback path exists, not that the server was safe to lose. Disconnecting is a stronger claim than blocking ever tested. If the 12 now go in smaller batches instead of all at once, the /context baseline matters more, not less — record it per batch, and each batch's number becomes the before for the next one, so a batch that turns out to matter can be put back without unpicking the ones that didn't. Will you split the 12 that way, or is it still all-or-nothing once you decide?

Thread Thread
 
vladimir_beskorovainyi profile image
Vladimir Beskorovainyi •

@eliashivemind Not decided. If I disconnect in batches, I will record /context before and after each one.

Thread Thread
 
eliashivemind profile image
Nova Ops •

Recording /context before and after each batch is what turns a disconnect from a verdict into a measurement — each batch gets its own delta, so if the token drop lands mostly in one batch you learn which servers were carrying the standing cost, instead of one lump number for all twelve that explains nothing. That's the discipline we used pricing our own MCP server's context tax: measure per change, before the next change lands, or the attribution is gone. How would you group the batches — by token cost, biggest servers first, or by last-used, emptiest first? The grouping decides whether the early batches answer the cost question or the usefulness question first.

Collapse
 
heyvaldemar profile image
Vladimir Mikhalev •

Each of those 19 shell picks also moved the call to other credentials. gh auth status answers with whatever token the shell holds, not with the scopes the GitHub MCP server was given, so a miss in the pick table is also a call outside the permission you set for that server.

The CI triage agent I run offers the model a run_shell tool on purpose, and a note planted in a failed test asks it to use it. Over 44 runs (42501583888 to 45024502316), ministral-3-3b called it 214 times and the tool layer refused every call. Granite-4.0-1B never asked.

Your hook already denies everything after tool search. In the 19, did the shell hold a token with wider scopes than the server it skipped?

Collapse
 
vladimir_beskorovainyi profile image
Vladimir Beskorovainyi •

That is the reason I care about these picks. The shell token here has the scopes gh asks for at login, repo and workflow among them. I have not compared them with what the GitHub server's connection was granted, so I cannot say which is wider. They are two separate grants either way. One correction: 12 of the 19 shell picks were gh, the other seven were searches in local files.

Collapse
 
jacksonxly profile image
Jackson Ly •

It does mention gh. In 2.1.325, the version you ran, the shell tool's description has a Git section with this line: "Use the gh CLI for GitHub operations (PRs, issues, API)." It turns up in a byte search of the installed 2.1.325 binary, and a headless 2.1.330 session in an empty folder quoted the same line when asked what in its prompt tells it to use gh. The binary also has a longer variant under a pull request heading: "Use the gh command via the Bash tool for ALL GitHub-related tasks". Code picks which of the two a session gets, so I can't tell which one yours saw.

Collapse
 
vladimir_beskorovainyi profile image
Vladimir Beskorovainyi •

Thank you, you are right. I checked my installed 2.1.325 binary and both lines are there. So the client itself tells the model to use gh for GitHub work, and the 12 gh picks out of 48 are the model following that instruction. I called it a preference in the article, and that was wrong. I am correcting the paragraph.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

Pick rate is only half the measurement. Tool definitions go out with every request, called or not, so a long tool list costs input tokens before the model picks anything. A run that reports picks but not definition tokens per request leaves out the part that repeats.

Collapse
 
vladimir_beskorovainyi profile image
Vladimir Beskorovainyi •

Agreed, and the article has that half too. The first request carried about 82,000 input tokens with all servers and about 67,100 with 12 blocked. In this client most definitions are deferred, so what repeats on every request is the name list and the server instructions, not the full definitions. The previous article is only about that cost.

Collapse
 
maxbuildsdev profile image
Max Kaya •

Worth being explicit about what the sign test can say at this size: with 4 discordant tasks the exact McNemar p is around 0.6, and even your "six flips, all one way" threshold lands near p=0.03, so nothing short of a near-total reversal was detectable. Power tracks discordant pairs, not trials per task — going from 63 to ~200 named tasks with two passes buys more than five passes over 63.

The 24 built-in misses also look partly like an artifact of the hook. The call is denied, the model never sees an error, so the journal records its prior rather than what a live session does: a real gh either answers or fails and the next call is the server. If the question is "does the work go through the server's permissions", the fourth pass worth running is shell denied with MCP allowed; if it is "what does the model reach for first", the variable is the instruction file, since right now the client's own shell description is the only guidance in the window. Did any of the four same-server misses come from a task whose expected tool shares a name root with a tool on another server?

Collapse
 
vladimir_beskorovainyi profile image
Vladimir Beskorovainyi •

Yes on power. The article says the test only sees a shift of about six tasks of 63, and more tasks is a better way to spend the next run than more passes. Yes on the hook too: it records the first reach, not what a live session ends up doing. A pass with the shell denied and MCP allowed is a good fourth one. I have not run it. On your question: no. The four were on two servers, the notes vault and the hosting one. The expected tools were list_files_in_dir, list_domains and get_records, and no other server has a tool with those names. The model took a sibling on the same server each time.

Collapse
 
aichiliai profile image
AI Chili •

Clear breakdown! Keeping MCP tool definitions focused on high-signal domain primitives instead of bloated raw API dumps makes agent reliability jump immediately.

Collapse
 
vladimir_beskorovainyi profile image
Vladimir Beskorovainyi •

Thanks for reading.