DEV Community

bredmond1019
bredmond1019

Posted on • Originally published at brandon-redmond.vercel.app

Building Intelligent AI Agents with Memory: A Complete Guide

Have you ever wondered why ChatGPT forgets your name after every conversation? Or why your AI assistant can't remember that you prefer concise answers? The answer lies in a fundamental limitation: most AI applications today are stateless.

But what if your AI agents could remember? What if they could build relationships, learn from failures, and adapt to user preferences over time? This is the promise of agent memory systems, and it's about to transform how we build AI applications.

Why Memory Matters: The Intelligence Connection

As Richmond Alake points out in his brilliant talk, if AI is meant to mimic human intelligence, and human intelligence is fundamentally tied to memory, then it's a "no-brainer" that our agents need memory too.

Think about the most intelligent people you know. What makes them stand out? It's their ability to:

  • Recall relevant information at the right time
  • Learn from past experiences
  • Build upon previous knowledge
  • Maintain context across interactions

These are exactly the capabilities we need in our AI agents.

The Memory Spectrum: From Chatbots to Autonomous Agents

The evolution of AI applications has been rapid:

  1. Chatbots (2022): Simple Q&A interfaces
  2. RAG Systems (2023): Domain-specific knowledge integration
  3. Tool-Using Agents (2024): LLMs with function calling
  4. Memory-Enabled Agents (Now): Stateful, relationship-building systems

Each evolution has added capabilities, but memory is the key to unlocking true agent intelligence.

Understanding Agent Memory Types

Let's dive deep into the different types of memory your agents need, with practical implementation examples for each.

1. Conversational Memory

The most basic form - remembering what was said in a conversation.

interface ConversationMemory {
  conversationId: string;
  messages: Message[];
  timestamp: Date;
  metadata: {
    userId: string;
    sessionDuration: number;
    topicsSummary: string[];
  };
}

class ConversationMemoryManager {
  async store(message: Message, conversationId: string) {
    // Store in MongoDB with automatic indexing
    await this.db.collection('conversations').updateOne(
      { conversationId },
      { 
        $push: { messages: message },
        $set: { lastUpdated: new Date() },
        $inc: { messageCount: 1 }
      },
      { upsert: true }
    );
  }

  async retrieve(conversationId: string, limit: number = 10) {
    // Retrieve recent messages with context
    return await this.db.collection('conversations').findOne(
      { conversationId },
      { 
        messages: { $slice: -limit },
        projection: { messages: 1, metadata: 1 }
      }
    );
  }
}
Enter fullscreen mode Exit fullscreen mode

2. Entity Memory

Tracking information about people, objects, and concepts mentioned in conversations.

interface EntityMemory {
  entityId: string;
  type: 'person' | 'organization' | 'location' | 'concept';
  attributes: Record<string, any>;
  relationships: Relationship[];
  mentions: EntityMention[];
  lastUpdated: Date;
}

class EntityMemoryManager {
  async updateEntity(entityData: Partial<EntityMemory>) {
    // Use MongoDB's flexible schema to store diverse entity types
    const entity = await this.db.collection('entities').findOneAndUpdate(
      { entityId: entityData.entityId },
      {
        $set: { 
          ...entityData,
          lastUpdated: new Date()
        },
        $addToSet: { 
          mentions: { $each: entityData.mentions || [] }
        }
      },
      { upsert: true, returnDocument: 'after' }
    );

    // Update vector embeddings for semantic search
    if (entity) {
      await this.updateEntityEmbedding(entity);
    }

    return entity;
  }

  async findRelatedEntities(entityId: string, relationshipType?: string) {
    // Graph-like queries in MongoDB
    return await this.db.collection('entities').find({
      'relationships.targetId': entityId,
      ...(relationshipType && { 'relationships.type': relationshipType })
    }).toArray();
  }
}
Enter fullscreen mode Exit fullscreen mode

3. Episodic Memory

Remembering sequences of events and experiences.

interface Episode {
  episodeId: string;
  startTime: Date;
  endTime?: Date;
  events: Event[];
  outcome: 'success' | 'failure' | 'partial';
  learnings: string[];
  context: Record<string, any>;
}

class EpisodicMemoryManager {
  async recordEpisode(episode: Episode) {
    // Store complete interaction sequences
    await this.db.collection('episodes').insertOne({
      ...episode,
      // Add searchable summary
      summary: await this.generateEpisodeSummary(episode),
      // Vector embedding for similarity search
      embedding: await this.generateEmbedding(episode)
    });
  }

  async findSimilarEpisodes(currentContext: any, limit: number = 5) {
    // Use vector search to find similar past experiences
    const embedding = await this.generateEmbedding(currentContext);

    return await this.db.collection('episodes').aggregate([
      {
        $vectorSearch: {
          index: 'episode_embeddings',
          path: 'embedding',
          queryVector: embedding,
          numCandidates: 100,
          limit: limit
        }
      },
      {
        $project: {
          episodeId: 1,
          summary: 1,
          learnings: 1,
          score: { $meta: 'vectorSearchScore' }
        }
      }
    ]).toArray();
  }
}
Enter fullscreen mode Exit fullscreen mode

4. Procedural Memory

Storing learned procedures and skills - like how the cerebellum stores motor skills.

interface Procedure {
  procedureId: string;
  name: string;
  steps: ProcedureStep[];
  conditions: Condition[];
  successRate: number;
  executionCount: number;
  averageExecutionTime: number;
  optimizations: Optimization[];
}

interface ProcedureStep {
  action: string;
  parameters: Record<string, any>;
  expectedOutcome: any;
  fallbackActions: string[];
}

class ProceduralMemoryManager {
  async learnProcedure(executionTrace: ExecutionTrace) {
    const procedurePattern = this.extractPattern(executionTrace);

    await this.db.collection('procedures').updateOne(
      { 'pattern.hash': procedurePattern.hash },
      {
        $inc: { 
          executionCount: 1,
          totalExecutionTime: executionTrace.duration
        },
        $push: { 
          executions: {
            $each: [executionTrace],
            $slice: -100 // Keep last 100 executions
          }
        },
        $set: {
          successRate: { 
            $divide: [
              { $size: { $filter: { 
                input: '$executions',
                cond: { $eq: ['$$this.outcome', 'success'] }
              }}},
              { $size: '$executions' }
            ]
          }
        }
      },
      { upsert: true }
    );
  }

  async selectBestProcedure(goal: string, context: any) {
    // Find procedures that match the goal and context
    const candidates = await this.db.collection('procedures').find({
      'applicableGoals': goal,
      'conditions': { $all: this.extractConditions(context) }
    }).sort({ successRate: -1, executionCount: -1 }).limit(5).toArray();

    // Return the most reliable procedure
    return candidates[0];
  }
}
Enter fullscreen mode Exit fullscreen mode

5. Semantic Memory

General knowledge and facts about the world.

interface SemanticFact {
  factId: string;
  subject: string;
  predicate: string;
  object: string;
  confidence: number;
  sources: Source[];
  contradictions: Contradiction[];
  lastVerified: Date;
}

class SemanticMemoryManager {
  async storeFact(fact: SemanticFact) {
    // Check for contradictions before storing
    const contradictions = await this.findContradictions(fact);

    if (contradictions.length > 0) {
      fact.contradictions = contradictions;
      // Resolve contradictions based on source reliability
      fact = await this.resolveContradictions(fact, contradictions);
    }

    await this.db.collection('semantic_facts').insertOne({
      ...fact,
      // Create knowledge graph connections
      tripleHash: this.hashTriple(fact.subject, fact.predicate, fact.object),
      embedding: await this.generateFactEmbedding(fact)
    });
  }

  async queryKnowledge(query: string) {
    // Natural language query to structured search
    const structuredQuery = await this.parseQuery(query);

    return await this.db.collection('semantic_facts').find({
      $or: [
        { subject: { $regex: structuredQuery.subject, $options: 'i' } },
        { object: { $regex: structuredQuery.object, $options: 'i' } },
        { $text: { $search: query } }
      ],
      confidence: { $gte: 0.7 }
    }).toArray();
  }
}
Enter fullscreen mode Exit fullscreen mode

6. Working Memory

Short-term memory for current task execution.

interface WorkingMemory {
  sessionId: string;
  currentGoal: string;
  activeContext: Record<string, any>;
  stack: TaskStack[];
  scratchpad: Record<string, any>;
  attentionWeights: AttentionWeight[];
  expiresAt: Date;
}

class WorkingMemoryManager {
  private cache: Map<string, WorkingMemory> = new Map();

  async maintain(sessionId: string) {
    const memory = this.cache.get(sessionId) || await this.load(sessionId);

    // Implement attention mechanism
    memory.attentionWeights = this.calculateAttention(memory);

    // Prune irrelevant information
    memory.activeContext = this.pruneContext(
      memory.activeContext,
      memory.attentionWeights
    );

    // Persist important information to long-term memory
    if (this.shouldPersist(memory)) {
      await this.persistToLongTerm(memory);
    }

    this.cache.set(sessionId, memory);
  }

  calculateAttention(memory: WorkingMemory): AttentionWeight[] {
    // Implement recency, relevance, and frequency based attention
    return Object.entries(memory.activeContext).map(([key, value]) => ({
      key,
      weight: this.calculateWeight(key, value, memory.currentGoal),
      lastAccessed: new Date()
    }));
  }
}
Enter fullscreen mode Exit fullscreen mode

7. Persona Memory

Agent personality and behavioral patterns.

interface PersonaMemory {
  agentId: string;
  personality: PersonalityTraits;
  communicationStyle: CommunicationPreferences;
  expertise: string[];
  values: Value[];
  behavioralPatterns: Pattern[];
  adaptations: Adaptation[];
}

class PersonaMemoryManager {
  async initializePersona(agentConfig: AgentConfig) {
    const persona: PersonaMemory = {
      agentId: agentConfig.id,
      personality: {
        openness: agentConfig.personality?.openness || 0.7,
        conscientiousness: agentConfig.personality?.conscientiousness || 0.8,
        extraversion: agentConfig.personality?.extraversion || 0.6,
        agreeableness: agentConfig.personality?.agreeableness || 0.8,
        emotionalStability: agentConfig.personality?.emotionalStability || 0.9
      },
      communicationStyle: {
        formality: agentConfig.style?.formality || 'professional',
        verbosity: agentConfig.style?.verbosity || 'balanced',
        humor: agentConfig.style?.humor || 'occasional',
        empathy: agentConfig.style?.empathy || 'high'
      },
      expertise: agentConfig.expertise || [],
      values: agentConfig.values || [],
      behavioralPatterns: [],
      adaptations: []
    };

    await this.db.collection('personas').insertOne(persona);
    return persona;
  }

  async adaptPersona(agentId: string, feedback: UserFeedback) {
    // Learn from user preferences
    const adaptation = this.analyzeAdaptation(feedback);

    await this.db.collection('personas').updateOne(
      { agentId },
      {
        $push: { 
          adaptations: {
            ...adaptation,
            timestamp: new Date(),
            confidence: this.calculateConfidence(adaptation)
          }
        },
        $set: {
          // Gradually adjust personality based on successful interactions
          'personality.openness': { 
            $avg: ['$personality.openness', adaptation.suggestedOpenness] 
          }
        }
      }
    );
  }
}
Enter fullscreen mode Exit fullscreen mode

8. Toolbox Memory

Dynamic tool discovery and selection.

interface Tool {
  toolId: string;
  name: string;
  description: string;
  inputSchema: JSONSchema;
  outputSchema: JSONSchema;
  examples: Example[];
  performanceMetrics: PerformanceMetrics;
  tags: string[];
  capabilities: string[];
}

class ToolboxMemoryManager {
  async registerTool(tool: Tool) {
    // Store tool with searchable metadata
    await this.db.collection('tools').insertOne({
      ...tool,
      embedding: await this.generateToolEmbedding(tool),
      searchableText: `${tool.name} ${tool.description} ${tool.tags.join(' ')}`
    });
  }

  async selectTools(task: string, context: any, maxTools: number = 5) {
    // Intelligent tool selection based on task requirements
    const taskEmbedding = await this.generateEmbedding(task);

    const relevantTools = await this.db.collection('tools').aggregate([
      {
        $vectorSearch: {
          index: 'tool_embeddings',
          path: 'embedding',
          queryVector: taskEmbedding,
          numCandidates: 50,
          limit: maxTools * 2
        }
      },
      {
        $match: {
          // Filter by performance metrics
          'performanceMetrics.successRate': { $gte: 0.8 },
          'performanceMetrics.averageExecutionTime': { $lte: 5000 }
        }
      },
      {
        $sort: {
          'performanceMetrics.successRate': -1,
          score: -1
        }
      },
      {
        $limit: maxTools
      }
    ]).toArray();

    return relevantTools;
  }

  async updateToolPerformance(toolId: string, execution: ToolExecution) {
    // Track tool performance for better selection
    await this.db.collection('tools').updateOne(
      { toolId },
      {
        $inc: {
          'performanceMetrics.totalExecutions': 1,
          'performanceMetrics.successfulExecutions': execution.success ? 1 : 0,
          'performanceMetrics.totalExecutionTime': execution.duration
        },
        $set: {
          'performanceMetrics.successRate': {
            $divide: [
              '$performanceMetrics.successfulExecutions',
              '$performanceMetrics.totalExecutions'
            ]
          },
          'performanceMetrics.averageExecutionTime': {
            $divide: [
              '$performanceMetrics.totalExecutionTime',
              '$performanceMetrics.totalExecutions'
            ]
          }
        }
      }
    );
  }
}
Enter fullscreen mode Exit fullscreen mode

Memory Management: The Core System

Building on these memory types, we need a comprehensive memory management system that handles the lifecycle of memories.

interface MemoryManager {
  generate(input: any): Promise<Memory>;
  store(memory: Memory): Promise<void>;
  retrieve(query: Query): Promise<Memory[]>;
  integrate(memories: Memory[]): Promise<IntegratedMemory>;
  update(memoryId: string, updates: Partial<Memory>): Promise<void>;
  forget(criteria: ForgetCriteria): Promise<void>;
}

class UnifiedMemoryManager implements MemoryManager {
  private memoryTypes: Map<string, BaseMemoryManager>;

  constructor(private db: MongoDB) {
    this.memoryTypes = new Map([
      ['conversation', new ConversationMemoryManager(db)],
      ['entity', new EntityMemoryManager(db)],
      ['episodic', new EpisodicMemoryManager(db)],
      ['procedural', new ProceduralMemoryManager(db)],
      ['semantic', new SemanticMemoryManager(db)],
      ['working', new WorkingMemoryManager(db)],
      ['persona', new PersonaMemoryManager(db)],
      ['toolbox', new ToolboxMemoryManager(db)]
    ]);
  }

  async generate(input: any): Promise<Memory> {
    // Analyze input to determine memory type
    const memoryType = await this.classifyMemoryType(input);
    const manager = this.memoryTypes.get(memoryType);

    // Extract relevant information
    const extractedInfo = await manager.extract(input);

    // Generate memory with metadata
    return {
      id: generateId(),
      type: memoryType,
      content: extractedInfo,
      metadata: {
        createdAt: new Date(),
        source: input.source,
        confidence: this.calculateConfidence(extractedInfo),
        tags: await this.generateTags(extractedInfo)
      }
    };
  }

  async retrieve(query: Query): Promise<Memory[]> {
    // Multi-modal retrieval across memory types
    const retrievalStrategies = [
      this.vectorSearch(query),
      this.keywordSearch(query),
      this.graphSearch(query),
      this.temporalSearch(query)
    ];

    const results = await Promise.all(retrievalStrategies);

    // Merge and rank results
    return this.rankMemories(
      this.mergeResults(results),
      query.context
    );
  }

  async integrate(memories: Memory[]): Promise<IntegratedMemory> {
    // Combine multiple memories into coherent context
    const integrated: IntegratedMemory = {
      memories: memories,
      summary: await this.summarizeMemories(memories),
      conflicts: this.detectConflicts(memories),
      gaps: this.identifyGaps(memories),
      recommendations: await this.generateRecommendations(memories)
    };

    // Resolve conflicts if any
    if (integrated.conflicts.length > 0) {
      integrated.resolvedConflicts = await this.resolveConflicts(
        integrated.conflicts
      );
    }

    return integrated;
  }

  async forget(criteria: ForgetCriteria): Promise<void> {
    // Implement forgetting mechanism (not deletion)
    const memoriesToForget = await this.db.collection('memories').find(
      criteria.filter
    ).toArray();

    for (const memory of memoriesToForget) {
      await this.db.collection('memories').updateOne(
        { _id: memory._id },
        {
          $inc: { 
            forgettingScore: criteria.forgettingRate 
          },
          $set: {
            lastForgettingUpdate: new Date()
          }
        }
      );
    }

    // Archive memories that exceed forgetting threshold
    await this.archiveHighlyForgottenMemories();
  }

  private async archiveHighlyForgottenMemories() {
    const threshold = 0.9;

    const forgottenMemories = await this.db.collection('memories').find({
      forgettingScore: { $gte: threshold }
    }).toArray();

    if (forgottenMemories.length > 0) {
      // Move to archive collection
      await this.db.collection('archived_memories').insertMany(
        forgottenMemories.map(m => ({
          ...m,
          archivedAt: new Date(),
          archiveReason: 'forgetting_threshold_exceeded'
        }))
      );

      // Remove from active memories
      await this.db.collection('memories').deleteMany({
        _id: { $in: forgottenMemories.map(m => m._id) }
      });
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Implementing Memory Signals

Richmond mentioned implementing memory signals like recall and recency. Here's how to build that:

interface MemorySignal {
  signalType: 'recency' | 'frequency' | 'importance' | 'emotion' | 'surprise';
  value: number;
  decay: DecayFunction;
}

class MemorySignalProcessor {
  calculateRecency(memory: Memory): number {
    const hoursSinceAccess = 
      (Date.now() - memory.lastAccessed.getTime()) / (1000 * 60 * 60);

    // Exponential decay with half-life of 24 hours
    return Math.exp(-0.693 * hoursSinceAccess / 24);
  }

  calculateFrequency(memory: Memory): number {
    // Logarithmic scaling to prevent over-weighting
    return Math.log2(memory.accessCount + 1) / 10;
  }

  calculateImportance(memory: Memory, context: any): number {
    // Context-dependent importance
    const baseImportance = memory.metadata.importance || 0.5;
    const contextRelevance = this.calculateContextRelevance(memory, context);
    const userPreference = this.getUserPreferenceScore(memory);

    return (baseImportance + contextRelevance + userPreference) / 3;
  }

  calculateEmotionalWeight(memory: Memory): number {
    // Memories with emotional content are better retained
    const emotions = memory.metadata.emotions || [];
    const emotionIntensity = emotions.reduce(
      (sum, emotion) => sum + emotion.intensity, 
      0
    ) / emotions.length;

    return Math.min(emotionIntensity * 1.5, 1);
  }

  calculateSurprise(memory: Memory, expectations: any): number {
    // Unexpected information is more memorable
    const expectedValue = this.getExpectedValue(memory.type, expectations);
    const actualValue = memory.content;

    const surprise = this.calculateKLDivergence(expectedValue, actualValue);
    return Math.tanh(surprise); // Normalize to [0, 1]
  }

  combineSignals(signals: MemorySignal[]): number {
    // Weighted combination of all signals
    const weights = {
      recency: 0.3,
      frequency: 0.2,
      importance: 0.25,
      emotion: 0.15,
      surprise: 0.1
    };

    return signals.reduce((total, signal) => {
      return total + (signal.value * weights[signal.signalType]);
    }, 0);
  }
}
Enter fullscreen mode Exit fullscreen mode

Production Considerations

1. Scalability

class ScalableMemoryArchitecture {
  constructor(
    private primaryDB: MongoDB,
    private cache: RedisCache,
    private vectorDB: VectorDatabase
  ) {}

  async setupSharding() {
    // Shard by agent ID for horizontal scaling
    await this.primaryDB.admin().command({
      shardCollection: 'memories.conversations',
      key: { agentId: 'hashed' }
    });

    // Shard by timestamp for time-series data
    await this.primaryDB.admin().command({
      shardCollection: 'memories.episodes',
      key: { timestamp: 1 }
    });
  }

  async setupIndexes() {
    // Compound indexes for common queries
    await this.primaryDB.collection('memories').createIndexes([
      { agentId: 1, timestamp: -1 },
      { userId: 1, memoryType: 1 },
      { 'metadata.tags': 1 },
      { 'content.text': 'text' } // Full-text search
    ]);

    // TTL index for automatic cleanup
    await this.primaryDB.collection('working_memory').createIndex(
      { expiresAt: 1 },
      { expireAfterSeconds: 0 }
    );
  }

  async implementCaching() {
    // Multi-layer caching strategy
    return {
      l1Cache: new InMemoryLRU({ maxSize: 1000 }), // Hot memories
      l2Cache: this.cache, // Distributed cache
      l3Storage: this.primaryDB // Persistent storage
    };
  }
}
Enter fullscreen mode Exit fullscreen mode

2. Privacy and Security

class SecureMemoryManager {
  async encryptSensitiveMemories(memory: Memory): Promise<EncryptedMemory> {
    // Identify and encrypt PII
    const piiFields = this.identifyPII(memory);

    for (const field of piiFields) {
      memory[field] = await this.encrypt(memory[field]);
    }

    return {
      ...memory,
      encrypted: true,
      encryptedFields: piiFields
    };
  }

  async implementAccessControl(agentId: string, userId: string) {
    // Role-based memory access
    const permissions = await this.getPermissions(agentId, userId);

    return {
      canRead: permissions.includes('read'),
      canWrite: permissions.includes('write'),
      canDelete: permissions.includes('delete'),
      memoryFilter: this.buildMemoryFilter(permissions)
    };
  }

  async auditMemoryAccess(access: MemoryAccess) {
    // Compliance and audit trail
    await this.db.collection('memory_audit').insertOne({
      ...access,
      timestamp: new Date(),
      ipAddress: access.request.ip,
      userAgent: access.request.userAgent
    });
  }
}
Enter fullscreen mode Exit fullscreen mode

3. Performance Optimization

class OptimizedMemoryRetrieval {
  async hybridSearch(query: string, context: any): Promise<Memory[]> {
    // Parallel execution of different search strategies
    const [
      vectorResults,
      keywordResults,
      graphResults
    ] = await Promise.all([
      this.vectorSearch(query, { limit: 20 }),
      this.keywordSearch(query, { fuzzy: true }),
      this.graphTraversal(context.entities, { depth: 2 })
    ]);

    // Intelligent result fusion
    return this.fuseResults(
      vectorResults,
      keywordResults,
      graphResults,
      { weights: [0.5, 0.3, 0.2] }
    );
  }

  async precomputeEmbeddings() {
    // Batch embedding generation
    const unprocessedMemories = await this.db.collection('memories').find({
      embedding: { $exists: false }
    }).limit(1000).toArray();

    const embeddings = await this.batchGenerateEmbeddings(
      unprocessedMemories.map(m => m.content)
    );

    // Bulk update
    const operations = unprocessedMemories.map((memory, index) => ({
      updateOne: {
        filter: { _id: memory._id },
        update: { $set: { embedding: embeddings[index] } }
      }
    }));

    await this.db.collection('memories').bulkWrite(operations);
  }
}
Enter fullscreen mode Exit fullscreen mode

Real-World Implementation: Customer Service Agent

Let's put it all together with a practical example:

class CustomerServiceAgent {
  private memoryManager: UnifiedMemoryManager;
  private llm: LLMProvider;

  async handleCustomerInteraction(
    message: string,
    customerId: string,
    sessionId: string
  ) {
    // 1. Load relevant memories
    const memories = await this.loadCustomerContext(customerId, sessionId);

    // 2. Update working memory
    await this.updateWorkingMemory(sessionId, {
      currentMessage: message,
      customerId,
      timestamp: new Date()
    });

    // 3. Extract entities and update entity memory
    const entities = await this.extractEntities(message);
    await this.updateEntityMemories(entities, customerId);

    // 4. Check for similar past interactions
    const similarEpisodes = await this.findSimilarInteractions(
      message,
      memories.episodic
    );

    // 5. Generate response with full context
    const response = await this.generateContextualResponse(
      message,
      memories,
      similarEpisodes
    );

    // 6. Store the interaction
    await this.storeInteraction({
      customerId,
      sessionId,
      message,
      response,
      entities,
      sentiment: await this.analyzeSentiment(message),
      resolution: response.resolved ? 'resolved' : 'ongoing'
    });

    // 7. Update customer profile
    await this.updateCustomerProfile(customerId, {
      lastInteraction: new Date(),
      interactionCount: { $inc: 1 },
      topics: entities.map(e => e.type),
      satisfactionTrend: this.calculateSatisfactionTrend(memories)
    });

    return response;
  }

  async loadCustomerContext(customerId: string, sessionId: string) {
    // Parallel load all relevant memory types
    const [
      conversational,
      entity,
      episodic,
      semantic,
      preferences
    ] = await Promise.all([
      this.memoryManager.retrieve({
        type: 'conversation',
        filter: { customerId },
        limit: 10
      }),
      this.memoryManager.retrieve({
        type: 'entity',
        filter: { 'mentions.customerId': customerId }
      }),
      this.memoryManager.retrieve({
        type: 'episodic',
        filter: { customerId },
        limit: 5
      }),
      this.memoryManager.retrieve({
        type: 'semantic',
        filter: { 'metadata.customerId': customerId }
      }),
      this.loadCustomerPreferences(customerId)
    ]);

    return {
      conversational,
      entity,
      episodic,
      semantic,
      preferences,
      summary: await this.summarizeContext({
        conversational,
        entity,
        episodic,
        semantic,
        preferences
      })
    };
  }

  async generateContextualResponse(
    message: string,
    memories: any,
    similarEpisodes: Episode[]
  ) {
    const prompt = `
You are a helpful customer service agent with access to the following context:

CUSTOMER HISTORY:
${memories.summary}

SIMILAR PAST INTERACTIONS:
${similarEpisodes.map(e => `- ${e.summary}: ${e.outcome}`).join('\n')}

CUSTOMER PREFERENCES:
${JSON.stringify(memories.preferences, null, 2)}

CURRENT MESSAGE:
${message}

Provide a helpful, personalized response that:
1. Acknowledges any past interactions if relevant
2. Uses the customer's preferred communication style
3. Resolves their issue effectively
4. Maintains consistency with previous interactions
`;

    const response = await this.llm.generate(prompt);

    return {
      text: response,
      resolved: this.checkIfResolved(response),
      nextSteps: this.extractNextSteps(response),
      sentiment: await this.analyzeSentiment(response)
    };
  }
}
Enter fullscreen mode Exit fullscreen mode

The Future: Neuroscience-Inspired Architectures

Richmond's talk mentioned the collaboration between neuroscientists and engineers. Here's a glimpse of what's coming:

class NeuroscienceInspiredMemory {
  // Implement consolidation like human sleep cycles
  async consolidateMemories() {
    // Transfer important short-term memories to long-term storage
    const importantMemories = await this.selectForConsolidation();

    for (const memory of importantMemories) {
      // Strengthen connections (like synaptic strengthening)
      await this.strengthenMemoryConnections(memory);

      // Create abstract representations
      await this.createAbstractRepresentation(memory);

      // Update memory networks
      await this.updateMemoryNetworks(memory);
    }
  }

  // Implement replay mechanisms
  async replayMemories(context: any) {
    // Like hippocampal replay during rest
    const relevantMemories = await this.selectRelevantMemories(context);

    // Simulate replay to strengthen patterns
    for (const memory of relevantMemories) {
      await this.simulateReplay(memory);
      await this.updatePatternStrength(memory);
    }
  }

  // Implement predictive coding
  async predictiveProcessing(input: any) {
    // Generate predictions based on memory
    const predictions = await this.generatePredictions(input);

    // Compare with actual input
    const predictionError = this.calculatePredictionError(predictions, input);

    // Update models based on error
    if (predictionError > this.threshold) {
      await this.updatePredictiveModels(predictionError);
    }

    return {
      predictions,
      error: predictionError,
      surprisingElements: this.identifySurprises(predictions, input)
    };
  }
}
Enter fullscreen mode Exit fullscreen mode

Conclusion: Memory as the Foundation of Intelligence

As we've explored, memory isn't just a nice-to-have feature for AI agents—it's the foundation of intelligence itself. By implementing comprehensive memory systems, we can transform our stateless AI applications into intelligent agents that:

  • Build genuine relationships with users
  • Learn from experience and improve over time
  • Maintain context across interactions
  • Adapt to individual preferences
  • Make better decisions based on past outcomes

The tools and patterns we've covered—from MongoDB's flexible document model to vector search capabilities—provide everything you need to build production-ready memory systems today.

Remember Richmond's key insight: we're not just building AI, we're architecting intelligence. And intelligence without memory is like trying to navigate life with permanent amnesia.

Start small—implement conversational memory first. Then gradually add other memory types as your agents grow more sophisticated. Your users will notice the difference, and your agents will finally be able to build the lasting relationships that make AI truly valuable.

📺 Watch the Original Talk

This post is based on Richmond Alake's excellent presentation on "Architecting Agent Memory" at MongoDB.

Watch on YouTube →

Richmond works at MongoDB and created the open-source Memoripy library mentioned in the talk. Connect with him on LinkedIn for more insights on building memory systems for AI agents.

Top comments (0)