RUMBA

RUMBA: Russian-language benchmark for long-term conversational memory in LLMs

Researchers introduce RUMBA (Russian User Memory BenchmArk), a new benchmark targeting long-term conversational memory in LLMs with a focus on Russian-language dialogues. The benchmark provides timestamped user-assistant conversations with QA pairs requiring retrieval, combination, and reasoning across sessions, along with a fine-grained taxonomy covering semantic type, session scope, temporal reasoning, and explicitness of temporal expressions. An aligned English subset is also provided. Evaluations of contemporary memory systems and long-context models reveal distinct strengths and failure modes across memory mechanisms.