MemOps

MemOps: Benchmark for lifecycle memory operations in long-horizon LLM conversations

MemOps is a new benchmark that reframes conversational memory evaluation as a sequence of explicit lifecycle operations—remembering, forgetting, updating, reflecting—rather than downstream question-answering accuracy alone. Each memory event is represented with a structured trace specifying trigger, target, scope, state transition, and evidence, embedded into long task-oriented conversations via a controllable generation pipeline. Evaluation across long-context, retrieval-based, parametric, and managed-memory systems reveals that current systems fail in distinct, previously conflated ways: session-level retrieval outperforms turn-level retrieval, and long-context models are notably weak at reconstructing ordered memory-state trajectories. The work shifts long-term memory evaluation from black-box final-answer scoring toward interpretable, operation-level diagnosis.