DEV Community

Frank Guan for Google AI

Posted on Edited on

Why Your AI Agent Keeps Forgetting (and Getting Slower): 4 Memory Options and 3 Context Fixes

Comments highlight the New York to SF test

Most AI agents have two memory problems. They forget what a user told them last week, and every turn costs more than the one before.

Both come from the same place: how much of the conversation you send to the model, and what you keep between sessions.

In this episode of AI Agent Clinic, Google Cloud engineer Laxmi Harikumar and developer Miles Libbey fix both in Scribant, a voice to action agent Miles built in TypeScript with Agent Development Kit (ADK).

Which memory option should you use?

The episode compares 4 ways to give an agent memory, using one test. A user says "I'm in New York", later says "I moved to San Francisco", then asks "Where do I live now?"

Option What the model gets Tradeoff
Flat prompting The full chat history Easy to build, but tokens grow every turn
Vector RAG Every chunk that matches, old and new Saves tokens, but retrieval alone doesn't say which fact is current
Knowledge graph The current fact Tracks changes, but you build the extraction, schemas and update logic
Memory Bank in Gemini platform The current fact Extracts, deduplicates and updates facts for you in the background

RAG is still the right tool for searching a knowledge base. Memory is a different job: keeping facts about a user current as they change.

How do you keep context small in a streaming agent?

Scribant gets new transcript text every 2 seconds, so resending everything adds up fast. The episode adds 3 fixes, each for a different time span:

  1. Rolling window (this turn): the intent classifier reads only the last 40 words.
  2. Rolling summary (this session): a running summary stands in for everything said so far.
  3. Memory Bank (across sessions): only the extracted facts are stored and recalled later.

What did the demo measure?

On one short test conversation in the episode (about 200 spoken words):

  • The 40 word window used about 86% fewer classifier tokens than sending the whole speech stream.
  • The rolling summary came to 92 tokens.
  • Memory Bank stored about 24 tokens of facts, where ingesting the whole conversation would have taken about 1,400.

These are results from one demo, not benchmarks. Treat 40 words as a starting point and tune it to how your users talk. The token tracking code is in the repo (token-telemetry.ts) if you want to measure your own agent.

Jump to a section

  • 0:00 Intro
  • 1:51 Live demo: spoken thoughts become calendar events and tasks
  • 2:44 Architecture: the 3 stage ADK pipeline and the execution gate
  • 4:21 Memory options: flat prompting, RAG and knowledge graphs
  • 5:56 Wiring up Memory Bank in ADK
  • 9:01 The 40 word rolling window
  • 12:07 Dashboard results
  • 14:41 Takeaways

Code and docs

How do you handle memory in your own agents: full history, RAG, or something else? Tell us in the comments.

Top comments (4)

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The memory-versus-retrieval split is the load-bearing distinction — RAG finds facts, but only a memory layer can say which fact is current, and the New York-to-San Francisco test makes that concrete. The 24 tokens of stored facts versus 1,400 for the whole conversation is the number that sells it. What happens when Memory Bank extracts a wrong fact — is there a correction path, or does the stale fact just keep winning until someone notices?

Collapse
 
ayushdhanani369 profile image
Ayush Dhanani •

I use something else, and your RAG point is why. I build Aladdin, a job agent. A fresh worker reads the actual form for each application, instead of carrying a long chat history into it. It only uses facts from the user's own profile that are clearly known. If a question is reworded or the answer is not clear, it asks the user instead of reusing an old answer. That keeps a corrected fact from coming back as a stale one.

Collapse
 
izgorodin profile image
Edward Izgorodin •

The New York to San Francisco test checks one thing: that the newest value wins. Two more lines in the same script catch the stores that pass it for the wrong reason. After the move, ask "Where did I live before San Francisco?": a store that overwrote the fact has nothing to say, while one that kept both facts, with the date the first one stopped being true, answers New York. Then say "I am moving to Seattle in March" and ask "Where do I live now?": an updater that treats the latest statement as the current fact answers Seattle months early. The first line tests history, the second tests the difference between when a fact was said and when it becomes true.

Collapse
 
ba796 profile image
brent alexis •

Interesting to see the rolling summary in there, because I live with it from the user side. I run my side project with an AI assistant, and when a long session runs out of room it gets summarized and carries on. That keeps the work going, but small details can drop out - an exact number, or a rule I gave it hours earlier.

What helped me was keeping my standing rules in a separate notes file that gets read at the start of every session. Things like "I do all the commits myself" or "check the live page after every push". They never live in the summary, so they can't get squashed. Pretty much your Memory Bank idea, just done by hand.

Did the demo check what the rolling summary leaves out after a long session, not just how small it gets?

This reply was drafted with AI help, edited by me.