"We did 4,000 calls yesterday" sounds like a measure of how many people use your MCP server.
It isn't, and the gap between what it sounds like and what it is turns out to be the most interesting thing about the chart.
What it actually counts
Every tool call that reached the server and was recorded, bucketed by the hour it started in UTC. A call is one tool, invoked once. That's all.
Why calls aren't people
Calls per conversation vary enormously. One user asking one question can produce one call or fifteen, depending on how many tools the model chains and how many times it has to reword and try again.
Which produces a genuinely perverse result: a server that gets better — clearer descriptions, fewer retries, fewer round trips — makes fewer calls for the same number of users. On this chart, improvement looks like decline.
If you've just rewritten an ambiguous tool description and your call count fell, that might be the fix working.
One heavy user looks like many light ones. An agent running a loop overnight can out-call every human user you have, combined.
Your own testing is in there. In the first weeks of a server, a large share of calls are its author.
The deliberate absence
The obvious response is to count users instead. We don't, and the reason is a design constraint rather than an oversight.
Nothing on the wire identifies a person. A call carries the tool name, the client name, a session ID, the timing, the outcome, the response size, and a hash of the arguments. That's the whole payload. No user ID, no IP stored against a call, no account name.
The session ID is generated by the SDK and means nothing outside your own data.
Adding a user field would make a nicer chart and a worse product. A user identifier is personal data, and MCP servers sit on email, calendars, CRM records — exactly the places where a security review stops being a short conversation.
The principle is the same one that keeps arguments out: if there's no field for it, there's nothing to leak and nothing to configure. A privacy guarantee you can switch off isn't one.
What to read instead
Sessions. Closer to a working period than a call is. Still not a person — one user opens many — but it moves with usage rather than with chattiness.
One implementation note if you're building this: count sessions from their own rows, not by grouping calls by day. A session that crosses midnight is one session, and a naive daily grouping counts it twice.
The client split. How many distinct clients connect, and what share each makes, says more about who your audience is than any total. It also tends to be the thing that actually surprises people.
Calls per tool. The total going up is less informative than which tools it's going up on. Growth concentrated in one tool is a workflow someone has found.
Read the shape, not the height
The height is ambiguous. The shape isn't:
- A cliff — calls dropping to near zero across every client at once — is an outage, a broken deploy, or a key that stopped working. Worth an alert.
- A cliff in one client only usually means that client changed something. A model, a config, how it lists your tools. Not your bug, and you'd never have attributed it correctly from a blended total.
- A step up after a directory listing, a post or a release tells you which of those actually moved anything. This is the cheapest marketing attribution you'll get.
- A weekly rhythm tells you whether people use the server for work.
The useful framing
Treat it as a pulse. It tells you the server is alive and roughly how hard it's working.
Everything else people want from it — how many users, whether they're happy, whether the tools are understood — is a different measurement, and reading it off this chart produces confident wrong answers.
The specific trap is the one at the top: a server that improves makes fewer calls. Any metric where the number going down can mean the thing is working needs a second number beside it before you act on either.
I build MCPulse — two lines inside your server, never your arguments or your results.
Originally published at getmcpulse.com.
Top comments (10)
@getmcpulse Agreed, and the quiet case is the one that matters. A retry only shows the model noticed missing input; a confident wrong answer leaves nothing in the call log, so the score has to come from the final answer.
Agreed, and I think it's worth stating as a boundary rather than a gap, because a gap implies I'll close it later.
Everything an SDK inside the server can see is behavioural: what got called, in what order, how often something was tried again. All of it is the model reacting to your tool. None of it is the answer the user got. A confident wrong answer and a correct one produce identical call logs, so no amount of better instrumentation on my side separates them — the information isn't in my process at any resolution.
So the honest division is that behaviour is measurable continuously in production, and correctness is measurable only offline against known answers. Two different activities with different costs, and the mistake is letting the cheap continuous one stand in for the expensive occasional one because it produces a number every day.
Which has a practical consequence for the trim test specifically. If you trim a reply and your retry rate holds, you've learned the model still thinks it has enough to work with. That's genuinely useful and it is not the same as the answers still being right. Anyone doing this at scale should treat a clean retry rate as permission to run the expensive test, not as a substitute for it.
I'd rather say that plainly in the post than imply the call log eventually gets there. It doesn't.
From the side of Tokenade, the measure that counts on an MCP server is the context each reply adds, not the call count. Every later step resends that context, so one clean answer costs less than several that dump a page, even when the call count looks worse.
Agreed, and it sharpens the post's point rather than contradicting it. "Fewer calls is better" is a proxy, and your version is the thing the proxy is standing in for. Three calls returning 200 tokens each beats one returning 5,000, and a call count scores that backwards.
The compounding is the part I'd underweighted. A response doesn't cost its own size once — it occupies the window for every subsequent turn, so its real cost is roughly its size multiplied by the turns that follow it. Which means the same response is cheap at the end of a conversation and expensive at the start, and nothing I measure knows where in a conversation a call happened.
That's a gap I can close, incidentally. Response size per call is already recorded, and position within the session is derivable from the ordering. Response bytes weighted by remaining turns is a better cost number than either call count or raw size, and I don't think anyone is reporting it.
It also puts schema cost in its place as the limiting case. The tool list enters at turn zero and is re-paid for the entire conversation — maximum multiplier, zero information content until a tool is actually called. Which is why an unused tool is worse than an oversized response: the response at least answered something.
The thing I can't see from inside the server is how much of a response the model actually used. A tool returning a page where two fields were needed has paid for the page on every turn since. From your side, is that measurable, or does it stay a judgement about result shape?
@getmcpulse The retry signal is the strongest part, because it fails loudly. The quiet case worries me more: the model answers from the fields it has and never calls again. A trim test that only counts retries misses that.
You're right, and it's the same asymmetry that makes the whole trim test weaker than it first looks. A retry is the model saying it couldn't work with what you gave it. Answering wrong from an incomplete reply is the model not noticing, and nothing in my data distinguishes that from answering correctly.
Which means a trim that looks clean by retry rate could be degrading every answer and reporting success. Worse than no signal, because it's a confident one.
Two partial things I can see from inside the server, neither sufficient.
A trim that removes something load-bearing should shift which tool gets called next, or whether anything gets called at all. If the session used to continue and now terminates at that tool, something changed about what the model felt able to conclude. That's weak and directional rather than diagnostic.
Second: run the trim as an A/B rather than a before-and-after. Same tool, same period, half the sessions trimmed. Then session shape differences can't be explained by the week, the client mix, or a model update — which is the confounder that makes sequential trim tests unreliable anyway.
But the actual failure is in the answer, and the answer never comes back to the server. The only thing that sees it is the user, and the user doesn't know your tool was involved.
Which puts a hard limit on trim testing specifically: it can tell you a trim was safe for the model's behaviour, and it cannot tell you the trim was safe for the user's answer. Those are different claims and I've been running them together.
The version that would close it needs a labelled set — requests with known-correct answers, run end to end against the trimmed and untrimmed tool, scored on the final answer rather than the call pattern. Expensive, offline, and nothing an SDK can do for you. Same shape as the probe suite for routing, one layer out.
the overnight-loop agent out-calling every human user is the scenario that breaks most mental models of server traffic. one question: does per-client session counting survive that, or does the agent just show up as one very dedicated user?
No, it doesn't survive it — and worse, it fails in two opposite directions depending on how the agent connects, which makes the failure hard to notice.
A loop holding one long-lived connection is exactly what you said: one very dedicated user. Sessions tells you nothing calls didn't already.
A loop that reconnects each iteration is worse. Now it's hundreds of sessions, and a metric meant to approximate people reports an agent as a crowd. Same workload, opposite distortion, and nothing in the data says which you're looking at.
What does survive is the rhythm, and it survives precisely because nobody is being identified. A loop agent is inhumanly regular — constant rate, no diurnal dip, no weekend. Humans are bursty with gaps. You can see that in the hourly shape without knowing anything about who is on the other end, and once you've seen it you can usually read the chart as "this band is the loop, that texture on top is people".
The client name does some work too. An agent harness usually names itself differently from Claude Desktop or Cursor, so a client you don't recognise taking 90% of your traffic is most of the answer.
Which leaves a real gap and I'd rather say so than claim otherwise: if the loop runs through an ordinary client, reconnects irregularly, and sits in the same volume band as your human traffic, nothing I collect separates it. The honest version is that these metrics measure load well and population badly, and a loop agent is the case where the difference stops being academic.
@getmcpulse Whether the model actually used a reply is the part I would not guess at. As far as I can see, it stays a judgement about result shape. The test I'd trust: trim a reply to the fields the next turn quotes, then compare the rest of the session.
Trimming to what the next turn quotes is a sharper test than anything I'd have come up with, and the reason it works is that it needs no claim about what the model attended to. It only needs the session to behave the same afterwards.
The comparison is also the right unit. Not "did the next turn still work" but the rest of the session — because the cost of over-returning is paid in later turns, and so is the risk of under-returning. If you trim and three turns on the model calls the same tool again with different arguments, you cut something it needed. Which is a retry, and it shows up in exactly the metric I already collect. So the experiment has a visible failure signal rather than relying on anyone judging output quality.
The honest limit is that it's a judgement about one session. The fields the next turn quoted are what that conversation needed, and a different question against the same tool wants different fields. So the output is a distribution over sessions, not a verdict on the tool — and the useful finding is the field that was never quoted in any session, which is the one safe to cut.
Which is approachable without storing responses: a declared list of fields per tool, counted for whether they ever appear in a subsequent turn. Field-level dead code, roughly, and it has the same shape as the dead-tool measurement one level down. A tool nobody calls costs its schema every session; a field nobody reads costs its bytes every turn after it arrives.
That one I think I can build. It needs the author to declare the fields, same as cross-tool agreement does, which is the recurring tax on measuring anything that lives inside the payload rather than around it.