Entity · paper

What do Reward Models Memorize?

paperactiveprovisionalwhat-do-reward-models-memorize--69d6a5b7·1 events·first seen 3d ago

Aliases: What do Reward Models Memorize?

More like this (12)

Process Reward Model Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design reward model Recalling Too Well: Sycophancy Evaluation and Mitigation in Memory-Augmented Models Reward Learning from Comparisons Holonomy Memory Reinforcement Learning rule-based reinforcement learning rewards Scaling Laws for Reward Model Overoptimization NAMESAKES: Probing Identity Memorization in Text-to-Image Models In-Context Reward Adaptation Gradient-Guided Reward Optimization Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learning

Recent events (1)

6arXiv · cs.CL·3d ago·source ↗

Study finds reward models memorize shortcuts and misallocate capacity, limiting context-dependent judgment

A new arXiv paper measures counterfactual memorization in discriminatively trained reward models (RMs) across two human preference datasets. The authors find that RMs misallocate memorization to easy high-margin pairs, learn dataset-specific shortcuts (e.g., model identity, user sampling strategy), and overgeneralize simple heuristics like response length and compliance. The findings suggest current discriminative RM training produces biased models that fail to judge response quality in context-dependent scenarios, with direct implications for RLHF pipelines.

Evaluation and Benchmarking Alignment and RLHF What do Reward Models Memorize?