paper
Generalization in offline RL: The structure is more important than the amount of pessimism
paperactiveprovisional
generalization-in-offline-rl-the-structure-is-more-important-than-the-amount-of-pessimism-e0cb1ac9·1 events·first seen 14h agoAliases: Generalization in offline RL: The structure is more important than the amount of pessimism
Co-occurring entities
More like this (12)
Pessimism's Paradox: Conservative Offline Training Amplifies Reward Hacking During Online Adaptation in Reasoning ModelsExpRL: Exploratory RL for LLM Mid-TrainingKL-regularized RLHierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode OutcomesRadial Suppression Accelerates Algorithmic Generalization: A Geometric Analysis of Delayed GeneralizationGeneralized LR ParsingGeneral Preference Reinforcement LearningThe Neutral Mask: How RLHF Provides Shallow Alignment while Leaving Partisan Structure Intact in a Large Language ModelIs One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL TrainingTuring-RLReinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMsUsing Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learning
Recent events (1)
Offline RL generalization depends on symmetry structure of pessimism, not its magnitude
A new arXiv preprint argues that successful generalization in offline reinforcement learning depends on whether the pessimistic value function respects the symmetries of the optimal solution, not on the degree of pessimism applied. The authors prove that a mildly pessimistic but non-symmetric value function can generalize worse than an overly pessimistic symmetric one, with implications for how data augmentation should be applied. They validate the theory empirically using IQL and CQL on a rotationally symmetric reacher environment, recommending a consistency loss during policy extraction over augmented dataset training.