Systematic evaluation of 12 agent memory systems from a data management perspective
A new arXiv preprint proposes an analytical framework decomposing agent memory into four core modules—representation/storage, extraction, retrieval/routing, and maintenance—and evaluates 12 representative memory systems across five benchmark workloads spanning 11 datasets. The study finds no single architecture dominates across scenarios; effectiveness depends on alignment between memory structure and workload bottleneck. Fine-grained ablation studies quantify effects on retrieval precision, update correctness, and long-horizon stability, and reveal that localized maintenance is more cost-efficient than global reorganization. Code is publicly released.
Related guides (3)
Related events (8)
GitOfThoughts: Git-based agent memory substrate with sobering findings on memory utility for novel problems
Researchers introduce GitOfThoughts, a system that stores LLM reasoning trees as git repositories, enabling replayable, auditable, and mergeable agent memory at low engineering cost. Across five memory substrates (none, markdown, vector, graph, git), two benchmarks, and two model scales with pre-registered replications, the paper finds that no memory format reliably improves accuracy on novel problems. Memory only helps above a 'copyability threshold' (similarity >~0.8), where retrieved cases are near-duplicates of the current problem — and even then, the gain is answer retrieval rather than method transfer. The paper also documents a retracted result and refuted hypothesis, modeling a rigorous evaluation standard.
MEMPROBE: Benchmark for auditing long-term agent memory via hidden user-state recovery
MEMPROBE is a new benchmark that evaluates long-term memory in LLM agents by treating memory as an auditable artifact rather than measuring it only through downstream task performance. After a memory-equipped agent assists simulated users across a trajectory of tasks, the benchmark attempts to reconstruct a hidden, taxonomy-anchored user-state bank from the agent's memory store. Testing across 5 memory systems and 50 simulated users with 31 hidden dimensions each, the authors find that task completion and memory recovery are largely independent capabilities — task success nearly saturates even for memoryless baselines, while structured user-state recovery remains moderate (~0.6) and degrades under top-k retrieval constraints.
agentmemory: Persistent Memory for AI Coding Agents
agentmemory is an open-source TypeScript library providing persistent memory for AI coding agents, designed based on real-world benchmarks. The repository has accumulated 13,772 total stars with a notable single-day gain of 1,626 stars, indicating strong community traction. It targets the agent tool ecosystem by addressing memory continuity across coding agent sessions.
LongMINT: Benchmark for Evaluating Memory Under Multi-Target Interference in Long-Horizon Agent Systems
LongMINT is a new benchmark designed to evaluate memory-augmented agents in realistic long-horizon settings where information is repeatedly updated and interferes across memories. It contains 15.6k QA pairs over contexts averaging 138.8k tokens (up to 1.8M tokens), spanning domains including state tracking, multi-turn dialogue, Wikipedia revisions, and GitHub commits. Evaluation of 7 representative systems—including vanilla long-context LLMs, RAG, and memory-augmented agent frameworks—reveals consistently low average accuracy of 27.9%, with performance particularly degraded on multi-target aggregation tasks and when earlier facts are revised by subsequent context. The analysis identifies retrieval and memory construction as the primary bottlenecks.
Infini Memory: Topic-structured persistent memory architecture for long-term LLM agents
Researchers propose Infini Memory, a persistent memory architecture for LLM agents that organizes memory as topic-structured documents rather than isolated records or summaries. New observations are staged in a buffer and periodically consolidated, while retrieval uses iterative agentic tool calls instead of a single lookup step. The system achieves 64.7% on MemoryAgentBench, with ablations showing complementary gains from topic-structured maintenance and iterative evidence inspection.
EvoArena benchmark and EvoMem memory paradigm for LLM agents in dynamic environments
Researchers introduce EvoArena, a benchmark suite that evaluates LLM agents in dynamic environments by modeling changes as progressive update sequences across terminal, software, and social domains. Alongside it, they propose EvoMem, a patch-based memory paradigm that records memory evolution as structured update histories to help agents reason about environmental change. Current agents score only 39.6% average accuracy on EvoArena, while EvoMem yields consistent gains on EvoArena and also improves performance on GAIA and LoCoMo benchmarks. The work highlights a significant gap between static-benchmark performance and real-world dynamic deployment requirements.
ENPMR-Bench: Benchmarking Proactive Memory Retrieval for Emotional Support Agents
This paper introduces ENPMR-Bench, a benchmark for evaluating Emotional Need-aware Proactive Memory Retrieval in memory-augmented language agents deployed for emotional support applications. The benchmark includes over 1,800 memory-augmented dialogues grounded in Maslow's hierarchy of needs, with structured mappings between emotional needs and supportive memory types. Experiments show that both embedding-based and LLM-driven retrieval paradigms fall significantly short of golden memory conditions on empathy scores, and while chain-of-thought prompting helps, a substantial performance gap remains. The work highlights a systematic gap in current agent memory systems when applied to affective rather than purely factual retrieval tasks.
VisualMem: Personal Visual Memory Benchmark and Architecture for Personalized AI Agents
This paper introduces a benchmark and hybrid architecture (VisualMem) for personal visual memory in long-term AI agent memory systems. The work addresses a gap in existing text-centric memory systems by capturing both explicit evidence (recurring user-associated entities) and implicit evidence (latent user facts from visual/multimodal cues) from images. VisualMem augments a text-memory backend with a structured personal visual memory module that uses conversational context to resolve identity, ownership, and durable user facts. Experiments show VisualMem substantially outperforms prior memory systems on the new benchmark while remaining competitive on standard text-memory benchmarks.