5arXiv cs.CL (Computation and Language)·
Setoka benchmark evaluates hierarchical user understanding in personalized agents
Researchers introduce Setoka, a benchmark for evaluating memory-augmented personalized agents across four levels of user understanding: semantic memory, episodic memory, behavior patterns, and personality traits. The benchmark uses a psychometrics-based pipeline to synthesize privacy-preserving heterogeneous user data, and evaluates 3 language models combined with 5 memory systems. Results show current systems handle semantic retrieval reasonably well but degrade significantly on higher-order tasks requiring cross-source integration and abstraction over long-term behavior, highlighting a gap in existing memory architectures.
Related guides (2)
Related events (8)
MEMPROBE: Benchmark for auditing long-term agent memory via hidden user-state recovery
MEMPROBE is a new benchmark that evaluates long-term memory in LLM agents by treating memory as an auditable artifact rather than measuring it only through downstream task performance. After a memory-equipped agent assists simulated users across a trajectory of tasks, the benchmark attempts to reconstruct a hidden, taxonomy-anchored user-state bank from the agent's memory store. Testing across 5 memory systems and 50 simulated users with 31 hidden dimensions each, the authors find that task completion and memory recovery are largely independent capabilities — task success nearly saturates even for memoryless baselines, while structured user-state recovery remains moderate (~0.6) and degrades under top-k retrieval constraints.
Benchmark Agent: Autonomous system for end-to-end benchmark construction
Researchers introduce Benchmark Agent, a fully autonomous agentic system that orchestrates the complete benchmark construction pipeline — from query analysis and subtask design to data annotation and quality control. The system was used to produce 15 benchmarks spanning text understanding, multimodal understanding, and domain-specific reasoning, with evaluation via human judges, LLM-as-a-judge, and consistency checks. The work addresses two persistent problems in the field: the labor intensity of benchmark creation and rapid performance saturation after release. Code and a demo will be publicly released.
M³Exam: Benchmark for Multimodal Memory in Realistic User-Agent Interactions
Researchers introduce M³Exam, a query-centric multimodal conversational memory benchmark designed to evaluate language agents on realistic user-agent interactions, including cross-modal grounding and implicit information inference. Existing benchmarks are critiqued for assuming sparse visuals and human-human interaction formats. The paper also proposes M³Proctor, a companion memory method that detects query modality bias and retrieves raw visual sources on demand, achieving 13% accuracy improvement while reducing index-construction time and retrieved tokens by over 70%.
ENPMR-Bench: Benchmarking Proactive Memory Retrieval for Emotional Support Agents
This paper introduces ENPMR-Bench, a benchmark for evaluating Emotional Need-aware Proactive Memory Retrieval in memory-augmented language agents deployed for emotional support applications. The benchmark includes over 1,800 memory-augmented dialogues grounded in Maslow's hierarchy of needs, with structured mappings between emotional needs and supportive memory types. Experiments show that both embedding-based and LLM-driven retrieval paradigms fall significantly short of golden memory conditions on empathy scores, and while chain-of-thought prompting helps, a substantial performance gap remains. The work highlights a systematic gap in current agent memory systems when applied to affective rather than purely factual retrieval tasks.
SovereignPA-Bench: A benchmark for evaluating user sovereignty in personal AI agents
Researchers introduce SovereignPA-Bench, an executable benchmark designed to evaluate personal AI agents on dimensions beyond task completion, including privacy preservation, consent compliance, evidence grounding, manipulation resistance, and user burden. The benchmark tests 120 sovereignty stress scenarios across 4 model families and 8 policy baselines, generating 3,840 frozen-prompt trajectories with a blinded human audit component. Results show that full-sovereign scaffolding outperforms all baselines on sovereignty metrics while reducing privacy leakage and consent violations. The work argues that personal-agent evaluation must shift from task success toward consent-aware, evidence-grounded action.
Systematic evaluation of 12 agent memory systems from a data management perspective
A new arXiv preprint proposes an analytical framework decomposing agent memory into four core modules—representation/storage, extraction, retrieval/routing, and maintenance—and evaluates 12 representative memory systems across five benchmark workloads spanning 11 datasets. The study finds no single architecture dominates across scenarios; effectiveness depends on alignment between memory structure and workload bottleneck. Fine-grained ablation studies quantify effects on retrieval precision, update correctness, and long-horizon stability, and reveal that localized maintenance is more cost-efficient than global reorganization. Code is publicly released.
OmniaBench: A broad diagnostic benchmark for general AI agents across 354 domains
OmniaBench is a new benchmark for evaluating general AI agents across diverse scenarios, spanning 90 level-1 and 354 level-2 domains derived from app stores, product documents, and web retrieval. The benchmark contains 1,431 tasks with single-turn and multi-turn formats, a ten-dimensional capability taxonomy, and eight atomic difficulty factors for fine-grained analysis. Even frontier models like Claude Sonnet 5 and GPT-5.6-Sol achieve only ~58% Overall Pass@1, revealing persistent weaknesses in planning, constraint maintenance, and adaptive correction. The work addresses a gap in existing agent benchmarks that tend to cover narrow tool ecosystems or interaction formats.
LongMINT: Benchmark for Evaluating Memory Under Multi-Target Interference in Long-Horizon Agent Systems
LongMINT is a new benchmark designed to evaluate memory-augmented agents in realistic long-horizon settings where information is repeatedly updated and interferes across memories. It contains 15.6k QA pairs over contexts averaging 138.8k tokens (up to 1.8M tokens), spanning domains including state tracking, multi-turn dialogue, Wikipedia revisions, and GitHub commits. Evaluation of 7 representative systems—including vanilla long-context LLMs, RAG, and memory-augmented agent frameworks—reveals consistently low average accuracy of 27.9%, with performance particularly degraded on multi-target aggregation tasks and when earlier facts are revised by subsequent context. The analysis identifies retrieval and memory construction as the primary bottlenecks.