setoka-9e3c8e3e·1 events·first seen Aliases: Setoka
Researchers introduce Setoka, a benchmark for evaluating memory-augmented personalized agents across four levels of user understanding: semantic memory, episodic memory, behavior patterns, and personality traits. The benchmark uses a psychometrics-based pipeline to synthesize privacy-preserving heterogeneous user data, and evaluates 3 language models combined with 5 memory systems. Results show current systems handle semantic retrieval reasonably well but degrade significantly on higher-order tasks requiring cross-source integration and abstraction over long-term behavior, highlighting a gap in existing memory architectures.