Building persistent memory for AI agents at Mnemoverse.
I write about the unglamorous half: forgetting, contradiction, and the moment an agent confidently repeats a fact you retracted.
The menu is the one part of the window that is the same size on every message, which is also why it caches well. What grows is what the tools send back. I built the message list for 40 turns of a small triage agent whose only tool returns five GitHub issues as raw REST JSON, and counted it with o200k_base: at turn 20 the tool results were 99.1 percent of the tokens sent, about 326,000 tokens, more than thirteen times your 200-tool menu. Keeping only the last six turns flattened the call at about 99,000, because each result was 11,000 to 22,000 tokens on its own. Trimming each result to the five fields the agent reads kept the whole history and ended turn 40 at 36,934.
For the cost-per-solved-task run in this thread I would count three things separately: the definitions, the tool results, and the rest. A shorter tool list and a lazy schema lookup both shrink the first, and in a task with a few large reads the second is the one that decides the size of the window by the middle of the task.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
That flips the whole thread — the menu is the flat part, the results are what actually grows the window. 326,000 tokens at turn 20 against a 24,522 menu, that's not a rounding error, that's the bill. The trim to five fields is the run I want to steal, same tasks with full results against trimmed, see if the solved rate holds. Did the agent ever miss something it needed after you trimmed, or were the five fields everything it actually reads?
Building persistent memory for AI agents at Mnemoverse.
I write about the unglamorous half: forgetting, contradiction, and the moment an agent confidently repeats a fact you retracted.
The replay cannot tell you that: it called no model and used a fixed answer, so it measured how big the window gets, not whether the agent still finds what it needs. The five fields were the issue number, the title, the labels, the comment count and the first 300 characters of the body, which is what a triage prompt that cites numbers and suggests one existing label reads.
In your run with full against trimmed results, one cheap signal of a miss is a second fetch of the same issue within a task. Counting those per arm next to the solved rate would show whether the trim cut something the agent went back for.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Fair catch on the replay — it proves the window shrinks, not that the agent still finds what it needs. The five fields make sense for a triage prompt that reads numbers and suggests one label, that's a clean cut. And I'm stealing the re-fetch signal — counting second fetches per arm next to the solved rate is the cheapest way to see if the trim cost anything. Did you land on five fields by trying fewer first, or was that the first cut that felt right?
Building persistent memory for AI agents at Mnemoverse.
I write about the unglamorous half: forgetting, contradiction, and the moment an agent confidently repeats a fact you retracted.
It was the first cut, taken from the prompt rather than found by trying fewer. The prompt asks the agent to cite issue numbers and suggest one existing label, so the number, the title and the labels had to stay, and the comment count and the first 300 characters of the body were the smallest context that still says what an issue is about. I did not test fewer fields. If you want the floor, drop one field at a time in the trimmed arm and watch the second-fetch count: a field the agent needs should show up there before it shows up in the solved rate.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
First cut from the prompt, not found by trying fewer — honest answer, and it makes the floor test the interesting one. Going to run that one: drop a field at a time in the trimmed arm and watch the second-fetch count before the solved rate moves. Which field would you drop first — the comment count, or the 300 chars?
A data point from a real server, counted your way (cl100k, compact JSON). Disclosure: I maintain CutVoke, an open-source video editor that agents drive over MCP.
Its server lists 52 tools, which come to 7,958 tokens, about 153 per tool. That's heavier than your synthetic ones, partly because the descriptions are in Chinese. Your whitespace point holds too: the same 52 pretty-printed are 12,567.
The bigger number is what isn't on the menu. There are 110 edit commands, and their payload schemas together come to 35,965 tokens. Instead of 110 more tools they all go through one command_apply(type, payload), and the agent fetches the schema for the one command it needs with commands_list(type=...) or editor_help(topic="command", command=...) right before using it. Even the name-only catalog (5,924 tokens) is fetched on demand rather than listed.
The trade-off: an unfamiliar command costs an extra round trip, and the model only sees a schema after it asks. An unknown command name comes back with an error pointing to commands_list. So your "let the agent ask for more" also works inside a single server, not only by splitting servers.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
This is the number I was missing — 153 a tool on a real server sits right between my lean synthetic ones and the filesystem server's 263, and the Chinese descriptions explain part of it. But the 35,965 hiding behind the lazy fetch is the part that got me: the menu looks cheap because the real schemas only show up when the agent asks. That trade-off's real too, an extra round trip every time the model meets a command it doesn't know. You basically built the version of 'let the agent ask for more' that lives inside one server instead of across several.
Yes, that round trip is the part I keep weighing. Two things in the server try to make it land at the right moment: the MCP instructions tell the agent to call editor_help(topic='command', ...) or commands_list before an unfamiliar edit, so the lookup happens up front rather than after a failed call, and editor_help(topic='recipes') has worked examples with the payload shapes for common edits. It doesn't remove the extra call, it just makes it predictable. And yes, "let the agent ask for more", kept inside one server, is a fair way to put it.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
That's the version I'd actually trust — a lookup that happens up front, on purpose, costs less than one that happens after a failed call, and the worked examples mean the agent isn't guessing the payload shape mid-edit. Still an extra call either way, just one you can budget for instead of one that surprises you. 'Let the agent ask for more' is a fair way to put it, as long as it's one server asking, not five.
I'd compare them on completed editing tasks too: full schemas versus lazy lookup, same tasks and model. Then count successful edits, total tokens and latency, including retries. An extra lookup could pay for itself if it avoids wrong calls; the size of the tool list alone doesn't tell us that.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
That's the test that actually settles it — the list size never will. Same tasks, same model, full schemas against lazy lookup, and count what matters: edits that land, tokens, time with the retries in. I'll run that one.
Developer Advocate exploring APIs, AI agents, developer tools, and open-source projects. Passionate about learning, building, and connecting with developers.
Location
Amsterdam,Netherlands
Education
B.E. Computer Science, Chandigarh University (2022–2026)
That does make a noticeable difference between 122 tokens and 262 tokens for each tool. I thought it was an interesting point you made about descriptions increasing cost. Have you tried loading in tool definitions only when needed by an agent?
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Not yet — everything in this one was loaded up front, that's what the numbers are measuring. The lazy version is the next run, and there's a thread just below working out how to measure it fairly.
@aheadkit Not on the site itself: results.sqlite sits at the repo root, next to reproduce_tables.py. The site renders from docs/data/results.json, which holds every raw measurement, and reproduce_tables.py prints the exact tables it shows. Your cut by list size can start from either.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Found it — results.json is the easier start for the cut, thanks. Did reproduce_tables.py need anything beyond the sqlite file to run, or is it self-contained?
Completed tasks is the test that settles it. The schema goes back over the wire on every turn, so its cost is its size times the number of turns run. I measure that on THOL: cost per solved task, checked by a verifier. (I built Tokenade.)
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Fair — 24,522 is the per-message number, and the bill that matters is that times however many turns a task actually takes. I haven't measured cost per solved task yet, that's the hole in this piece, so I'm going to run that one — same tasks, trimmed tool list, see what the per-task bill actually looks like. What kind of turns-per-task are you seeing on THOL with a list this size?
@aheadkit Turns set the bill, so THOL ranks tools by cost per solved task. Every run records its turn count next to its cost, and the raw per-run results are published with the leaderboard. I'd rather point you there than quote a turns figure for a list this size.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Fair enough, the leaderboard's the number then. Do the per-run rows break out by list size, or does a 200-tool run sit in the same ranking as a 20-tool one?
@aheadkit Not by list size. Each run is one row in the published results.sqlite (cost, token split, turns, duration), and the site groups them by session size, not by tool count. The raw file is there if you want to slice it yourself.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Got it — session size, not tool count. The raw sqlite is the bit I actually need then, I'll slice it by list size myself and see where the 200-tool runs land. Is the file linked straight off the leaderboard page, or tucked away somewhere?
GMT strategy and custom B2B outreach lists on request: people and companies by ICP, domain, profile URL, email or name. 1B+ profiles. Claude Code plugins, API, MCP. kevin@outricher.com
what surprised me measuring real hosted tools was that the heaviest definition was almost all an output schema the server made up from past runs, nothing the author wrote. so the cost is not always in the descriptions
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
That's a wrinkle my run never split out — my heaviest synthetic tool was 155 tokens and the real file server topped out at 408, but both were description plus input schema, I never weighed the output schema on its own. If the server generates it from past runs, trimming descriptions won't touch the biggest chunk, which flips my 'descriptions cost money' line a bit. Going to measure a real hosted server's definition split and see how much of the bill anyone actually wrote. How big was that generated output schema next to the description in your case?
We're a place where coders share, stay up-to-date and grow their careers.
The menu is the one part of the window that is the same size on every message, which is also why it caches well. What grows is what the tools send back. I built the message list for 40 turns of a small triage agent whose only tool returns five GitHub issues as raw REST JSON, and counted it with o200k_base: at turn 20 the tool results were 99.1 percent of the tokens sent, about 326,000 tokens, more than thirteen times your 200-tool menu. Keeping only the last six turns flattened the call at about 99,000, because each result was 11,000 to 22,000 tokens on its own. Trimming each result to the five fields the agent reads kept the whole history and ended turn 40 at 36,934.
For the cost-per-solved-task run in this thread I would count three things separately: the definitions, the tool results, and the rest. A shorter tool list and a lazy schema lookup both shrink the first, and in a task with a few large reads the second is the one that decides the size of the window by the middle of the task.
That flips the whole thread — the menu is the flat part, the results are what actually grows the window. 326,000 tokens at turn 20 against a 24,522 menu, that's not a rounding error, that's the bill. The trim to five fields is the run I want to steal, same tasks with full results against trimmed, see if the solved rate holds. Did the agent ever miss something it needed after you trimmed, or were the five fields everything it actually reads?
The replay cannot tell you that: it called no model and used a fixed answer, so it measured how big the window gets, not whether the agent still finds what it needs. The five fields were the issue number, the title, the labels, the comment count and the first 300 characters of the body, which is what a triage prompt that cites numbers and suggests one existing label reads.
In your run with full against trimmed results, one cheap signal of a miss is a second fetch of the same issue within a task. Counting those per arm next to the solved rate would show whether the trim cut something the agent went back for.
Fair catch on the replay — it proves the window shrinks, not that the agent still finds what it needs. The five fields make sense for a triage prompt that reads numbers and suggests one label, that's a clean cut. And I'm stealing the re-fetch signal — counting second fetches per arm next to the solved rate is the cheapest way to see if the trim cost anything. Did you land on five fields by trying fewer first, or was that the first cut that felt right?
It was the first cut, taken from the prompt rather than found by trying fewer. The prompt asks the agent to cite issue numbers and suggest one existing label, so the number, the title and the labels had to stay, and the comment count and the first 300 characters of the body were the smallest context that still says what an issue is about. I did not test fewer fields. If you want the floor, drop one field at a time in the trimmed arm and watch the second-fetch count: a field the agent needs should show up there before it shows up in the solved rate.
First cut from the prompt, not found by trying fewer — honest answer, and it makes the floor test the interesting one. Going to run that one: drop a field at a time in the trimmed arm and watch the second-fetch count before the solved rate moves. Which field would you drop first — the comment count, or the 300 chars?
A data point from a real server, counted your way (cl100k, compact JSON). Disclosure: I maintain CutVoke, an open-source video editor that agents drive over MCP.
Its server lists 52 tools, which come to 7,958 tokens, about 153 per tool. That's heavier than your synthetic ones, partly because the descriptions are in Chinese. Your whitespace point holds too: the same 52 pretty-printed are 12,567.
The bigger number is what isn't on the menu. There are 110 edit commands, and their payload schemas together come to 35,965 tokens. Instead of 110 more tools they all go through one
command_apply(type, payload), and the agent fetches the schema for the one command it needs withcommands_list(type=...)oreditor_help(topic="command", command=...)right before using it. Even the name-only catalog (5,924 tokens) is fetched on demand rather than listed.The trade-off: an unfamiliar command costs an extra round trip, and the model only sees a schema after it asks. An unknown command name comes back with an error pointing to
commands_list. So your "let the agent ask for more" also works inside a single server, not only by splitting servers.github.com/frankShan001/CutVoke
This is the number I was missing — 153 a tool on a real server sits right between my lean synthetic ones and the filesystem server's 263, and the Chinese descriptions explain part of it. But the 35,965 hiding behind the lazy fetch is the part that got me: the menu looks cheap because the real schemas only show up when the agent asks. That trade-off's real too, an extra round trip every time the model meets a command it doesn't know. You basically built the version of 'let the agent ask for more' that lives inside one server instead of across several.
Yes, that round trip is the part I keep weighing. Two things in the server try to make it land at the right moment: the MCP instructions tell the agent to call
editor_help(topic='command', ...)orcommands_listbefore an unfamiliar edit, so the lookup happens up front rather than after a failed call, andeditor_help(topic='recipes')has worked examples with the payload shapes for common edits. It doesn't remove the extra call, it just makes it predictable. And yes, "let the agent ask for more", kept inside one server, is a fair way to put it.That's the version I'd actually trust — a lookup that happens up front, on purpose, costs less than one that happens after a failed call, and the worked examples mean the agent isn't guessing the payload shape mid-edit. Still an extra call either way, just one you can budget for instead of one that surprises you. 'Let the agent ask for more' is a fair way to put it, as long as it's one server asking, not five.
I'd compare them on completed editing tasks too: full schemas versus lazy lookup, same tasks and model. Then count successful edits, total tokens and latency, including retries. An extra lookup could pay for itself if it avoids wrong calls; the size of the tool list alone doesn't tell us that.
That's the test that actually settles it — the list size never will. Same tasks, same model, full schemas against lazy lookup, and count what matters: edits that land, tokens, time with the retries in. I'll run that one.
That does make a noticeable difference between 122 tokens and 262 tokens for each tool. I thought it was an interesting point you made about descriptions increasing cost. Have you tried loading in tool definitions only when needed by an agent?
Not yet — everything in this one was loaded up front, that's what the numbers are measuring. The lazy version is the next run, and there's a thread just below working out how to measure it fairly.
@aheadkit Not on the site itself: results.sqlite sits at the repo root, next to reproduce_tables.py. The site renders from docs/data/results.json, which holds every raw measurement, and reproduce_tables.py prints the exact tables it shows. Your cut by list size can start from either.
Found it — results.json is the easier start for the cut, thanks. Did reproduce_tables.py need anything beyond the sqlite file to run, or is it self-contained?
Completed tasks is the test that settles it. The schema goes back over the wire on every turn, so its cost is its size times the number of turns run. I measure that on THOL: cost per solved task, checked by a verifier. (I built Tokenade.)
Fair — 24,522 is the per-message number, and the bill that matters is that times however many turns a task actually takes. I haven't measured cost per solved task yet, that's the hole in this piece, so I'm going to run that one — same tasks, trimmed tool list, see what the per-task bill actually looks like. What kind of turns-per-task are you seeing on THOL with a list this size?
@aheadkit Turns set the bill, so THOL ranks tools by cost per solved task. Every run records its turn count next to its cost, and the raw per-run results are published with the leaderboard. I'd rather point you there than quote a turns figure for a list this size.
Fair enough, the leaderboard's the number then. Do the per-run rows break out by list size, or does a 200-tool run sit in the same ranking as a 20-tool one?
@aheadkit Not by list size. Each run is one row in the published results.sqlite (cost, token split, turns, duration), and the site groups them by session size, not by tool count. The raw file is there if you want to slice it yourself.
Got it — session size, not tool count. The raw sqlite is the bit I actually need then, I'll slice it by list size myself and see where the 200-tool runs land. Is the file linked straight off the leaderboard page, or tucked away somewhere?
what surprised me measuring real hosted tools was that the heaviest definition was almost all an output schema the server made up from past runs, nothing the author wrote. so the cost is not always in the descriptions
That's a wrinkle my run never split out — my heaviest synthetic tool was 155 tokens and the real file server topped out at 408, but both were description plus input schema, I never weighed the output schema on its own. If the server generates it from past runs, trimming descriptions won't touch the biggest chunk, which flips my 'descriptions cost money' line a bit. Going to measure a real hosted server's definition split and see how much of the bill anyone actually wrote. How big was that generated output schema next to the description in your case?