CQL

Offline RL generalization depends on symmetry structure of pessimism, not its magnitude

A new arXiv preprint argues that successful generalization in offline reinforcement learning depends on whether the pessimistic value function respects the symmetries of the optimal solution, not on the degree of pessimism applied. The authors prove that a mildly pessimistic but non-symmetric value function can generalize worse than an overly pessimistic symmetric one, with implications for how data augmentation should be applied. They validate the theory empirically using IQL and CQL on a rotationally symmetric reacher environment, recommending a consistency loss during policy extraction over augmented dataset training.