How AI Agents Search Their Memory

In my last video, I covered how AI agents store memory. But storing it is only half the problem. In this one, we dig into how agents retrieve the right memory at the right time. I cover keyword search, semantic search, hybrid retrieval, and re-ranking, then dig into how OpenClaw implements all of it in practice using SQLite, BM25, and vector embeddings. --- *What you'll learn:* - How keyword search (grep, BM25) and semantic search (embeddings, vector similarity) each work -- and where they break down - Why hybrid search beats both approaches alone, and how weighted fusion and reciprocal rank fusion combine results - What re-ranking is, when to use it, and the cost/latency tradeoff - How a production agent memory system stores chunks, indexes them with FTS5 and sqlite-vec, and runs hybrid search in parallel - The two-step memory pattern: search then get, and why it keeps context windows lean - How incremental sync works: file watchers, content hashing, embedding caches, and full reindex triggers --- *Watch the full series:* - Part 1: How AI Agents Remember Things (the write path) → https://youtu.be/Seu7nksZ_4k - Part 2: How AI Agents Search Their Memory (this video) We use OpenClaw as the real-world case study -- it's a clean open-source implementation that shows these patterns in practice. --- *Resources mentioned:* - OpenClaw: https://github.com/openclaw/openclaw - SQLite FTS5 documentation: https://www.sqlite.org/fts5.html - sqlite-vec extension: https://github.com/asg017/sqlite-vec - OpenAI Embeddings API: https://platform.openai.com/docs/guides/embeddings - QMD by Toby Lutke: https://github.com/tobi/qmd - pgvector (Postgres vector extension, mentioned as an alternative): https://github.com/pgvector/pgvector - Pinecone (dedicated vector database, mentioned as an alternative): https://www.pinecone.io Get my full collection of Claude workflows and AI engineering notes: https://www.damiangalarza.com/newsletter?utm_source=youtube&utm_medium=video&utm_campaign=ai-agents-search-memory --- *Timestamps:* - 0:00 - The retrieval problem - 0:35 - Keyword search: grep and BM25 - 1:12 - Semantic search and how embeddings work - 2:04 - Vector databases and nearest neighbor search - 3:41 - Where semantic search breaks down - 4:05 - Hybrid search: combining keyword and semantic - 4:18 - Weighted fusion vs. reciprocal rank fusion - 5:01 - Re-ranking: trading speed for accuracy - 6:12 - OpenClaw's implementation walkthrough - 7:10 - Embedding providers and auto-selection - 8:08 - The SQLite schema: FTS5, sqlite-vec, and the embedding cache - 9:33 - The search pipeline step by step - 10:54 - Memory tools: search and get - 11:08 - Why the two-step pattern keeps context windows lean - 11:40 - The incremental sync and indexing pipeline - 12:44 - Wrap-up --- *About me:* I'm Damian Galarza, a software engineering leader and former CTO with 15+ years building SaaS products. I make practical AI engineering tutorials and share what I'm learning about agent architecture, LLM applications, and building production AI systems. *Connect:* - Newsletter: https://www.damiangalarza.com/newsletter?utm_source=youtube&utm_medium=video&utm_campaign=ai-agents-search-memory - LinkedIn: https://linkedin.com/in/dgalarza - Blog: https://www.damiangalarza.com #AIAgents #VectorSearch #RAG #SemanticSearch #OpenClaw