4arXiv cs.CL (Computation and Language)·
RUMBA: Russian-language benchmark for long-term conversational memory in LLMs
Researchers introduce RUMBA (Russian User Memory BenchmArk), a new benchmark targeting long-term conversational memory in LLMs with a focus on Russian-language dialogues. The benchmark provides timestamped user-assistant conversations with QA pairs requiring retrieval, combination, and reasoning across sessions, along with a fine-grained taxonomy covering semantic type, session scope, temporal reasoning, and explicitness of temporal expressions. An aligned English subset is also provided. Evaluations of contemporary memory systems and long-context models reveal distinct strengths and failure modes across memory mechanisms.
Related guides (2)
Related events (8)
M³Exam: Benchmark for Multimodal Memory in Realistic User-Agent Interactions
Researchers introduce M³Exam, a query-centric multimodal conversational memory benchmark designed to evaluate language agents on realistic user-agent interactions, including cross-modal grounding and implicit information inference. Existing benchmarks are critiqued for assuming sparse visuals and human-human interaction formats. The paper also proposes M³Proctor, a companion memory method that detects query modality bias and retrieves raw visual sources on demand, achieving 13% accuracy improvement while reducing index-construction time and retrieved tokens by over 70%.
IMLogic benchmark and RootMem framework target implicit logical memory retrieval for personalized LLMs
Researchers introduce IMLogic, a benchmark for evaluating implicit logical memory retrieval in long-dialogue personalized LLM scenarios, addressing gaps in existing semantic-similarity-based retrieval methods. They also propose RootMem, a plug-and-play framework that distills user histories into structured 'root memories' and uses an LLM-based router to activate logically relevant memories alongside semantic retrieval. Experiments show RootMem outperforms retrieval baselines and improves existing memory agents. The work targets a concrete weakness in current personalized LLM memory systems where logically critical memories lack semantic overlap with queries.
MemOps: Benchmark for lifecycle memory operations in long-horizon LLM conversations
MemOps is a new benchmark that reframes conversational memory evaluation as a sequence of explicit lifecycle operations—remembering, forgetting, updating, reflecting—rather than downstream question-answering accuracy alone. Each memory event is represented with a structured trace specifying trigger, target, scope, state transition, and evidence, embedded into long task-oriented conversations via a controllable generation pipeline. Evaluation across long-context, retrieval-based, parametric, and managed-memory systems reveals that current systems fail in distinct, previously conflated ways: session-level retrieval outperforms turn-level retrieval, and long-context models are notably weak at reconstructing ordered memory-state trajectories. The work shifts long-term memory evaluation from black-box final-answer scoring toward interpretable, operation-level diagnosis.
LongMINT: Benchmark for Evaluating Memory Under Multi-Target Interference in Long-Horizon Agent Systems
LongMINT is a new benchmark designed to evaluate memory-augmented agents in realistic long-horizon settings where information is repeatedly updated and interferes across memories. It contains 15.6k QA pairs over contexts averaging 138.8k tokens (up to 1.8M tokens), spanning domains including state tracking, multi-turn dialogue, Wikipedia revisions, and GitHub commits. Evaluation of 7 representative systems—including vanilla long-context LLMs, RAG, and memory-augmented agent frameworks—reveals consistently low average accuracy of 27.9%, with performance particularly degraded on multi-target aggregation tasks and when earlier facts are revised by subsequent context. The analysis identifies retrieval and memory construction as the primary bottlenecks.
MEMPROBE: Benchmark for auditing long-term agent memory via hidden user-state recovery
MEMPROBE is a new benchmark that evaluates long-term memory in LLM agents by treating memory as an auditable artifact rather than measuring it only through downstream task performance. After a memory-equipped agent assists simulated users across a trajectory of tasks, the benchmark attempts to reconstruct a hidden, taxonomy-anchored user-state bank from the agent's memory store. Testing across 5 memory systems and 50 simulated users with 31 hidden dimensions each, the authors find that task completion and memory recovery are largely independent capabilities — task success nearly saturates even for memoryless baselines, while structured user-state recovery remains moderate (~0.6) and degrades under top-k retrieval constraints.
TriggerBench: A benchmark for evaluating prospective memory in LLMs
Researchers introduce TriggerBench, a benchmark evaluating prospective memory (PM) in LLMs — the ability to spontaneously recall and act on latent constraints without explicit prompting. The benchmark spans five dimensions across daily assistant and professional workflow scenarios, and reveals that PM is substantially harder than retrospective memory, decaying sharply with context length while retrospective memory near-saturates at 100K tokens. Key findings include a precision-recall trade-off in PM, attentional fragility under concurrent requests, and a novel result that PM accuracy correlates with spare reasoning capacity as measured against AIME-2025 math performance.
BenCzechMark: A Benchmark for Evaluating LLM Czech Language Understanding
BenCzechMark is a new evaluation benchmark designed to assess large language model performance on Czech language tasks. The benchmark addresses the gap in non-English language evaluation, providing a structured way to measure LLM capabilities in Czech across multiple task types. Published on Hugging Face, it contributes to the growing ecosystem of multilingual and language-specific benchmarks.
TriviaRoomQA: Multilingual benchmark reveals LLM gaps in everyday popular-culture knowledge
Researchers introduce TriviaRoomQA, a multilingual benchmark with 3,300 parallel multiple-choice questions across six European languages and 5,340 French-only questions, covering 288 topics drawn from quiz and trivia contexts. Evaluating 30 open-weight LLMs (7B–70B parameters) from European, Asian, and North American providers, the study finds models perform well on history, geography, and math but substantially worse on popular-culture topics like celebrities, music, and movies. Performance also varies across languages for identical underlying questions, indicating factual knowledge access is language-dependent. The work highlights a knowledge gap not captured by existing saturated academic benchmarks.