In August my coding agent kept telling me that one kind of article on this site draws about four times as many comments as another. A week earlier I had corrected that number myself: with more data it was closer to two to one.
The old note was still in its memory. It sounded exactly as sure as the new one, and it kept winning, because nothing about it looked old.
That's the failure I find hardest to catch. A missing memory is loud: the agent asks, you answer, done. A stale one is quiet. It hands you a confident answer that stopped being true, and it reads like every correct answer it has ever given you.
I tried to fix it by ranking older notes lower, and that went badly.
So I'm collecting real cases for the next part of this series:
- What did your agent remember that had stopped being true? A config value, a renamed API, a decision your team reversed, a workaround for a bug that's long fixed.
- How did you notice? A failing build, a reviewer, a customer, luck?
- How long was it wrong before anyone saw it?
Short answers are welcome. The ones with a concrete example are the ones I'll learn the most from.
Top comments (2)
During release checks one of my agents read production at about 12:50 and found the default gate not set, then kept repeating "not set" all afternoon without checking again. I only noticed because I knew the gate was set and told it so. I still don't know whether the setting changed after 12:50 or it misread it the first time.
I build Aladdin, a job agent. The stale case I watch for is a form answer a user corrected once, then the agent keeps reusing the old version because it sounds confident.
My fix is boring on purpose. A fresh worker reads the actual page and form for each application, and it tailors only from the user's own facts. If the form cannot be read or an answer is unknown, it asks the user instead of guessing.
That does not remove stale facts entirely. It just makes the failure loud again, which is the part I want.