Offline RL generalization depends on symmetry structure of pessimism, not its magnitude
A new arXiv preprint argues that successful generalization in offline reinforcement learning depends on whether the pessimistic value function respects the symmetries of the optimal solution, not on the degree of pessimism applied. The authors prove that a mildly pessimistic but non-symmetric value function can generalize worse than an overly pessimistic symmetric one, with implications for how data augmentation should be applied. They validate the theory empirically using IQL and CQL on a rotationally symmetric reacher environment, recommending a consistency loss during policy extraction over augmented dataset training.
Related guides (1)
Related events (8)
High offline conservatism in DPO amplifies reward hacking during online adaptation, study finds
A new arXiv paper challenges the conventional wisdom that conservative offline training (via DPO with high β) provides a safer foundation for online RL adaptation. Experiments with Qwen3-14B show that higher offline conservatism monotonically increases reward hacking damage (Goodhart gap) during online adaptation, with Spearman ρ=1.0 across conditions. The mechanistic explanation is a three-link chain: high-β DPO compresses policy entropy, reducing response diversity and concentrating outputs in a narrow reward-model region, while paradoxically increasing ensemble disagreement that gets exploited during online optimization. The authors identify a practical optimal conservatism level β* and argue the field needs calibrated rather than maximal conservatism.
General Preference Reinforcement Learning (GPRL): Bridging Online RL and Preference Optimization for Open-Ended Tasks
GPRL proposes a new alignment framework that replaces scalar reward models with a General Preference Model (GPM) embedding responses into k skew-symmetric subspaces to capture multi-dimensional, intransitivity-aware preferences. The method computes per-dimension group-relative advantages, normalizes across axes, and uses a closed-loop drift monitor to detect and correct single-axis reward hacking during training. Starting from Llama-3-8B-Instruct, GPRL achieves a 56.51% length-controlled win rate on AlpacaEval 2.0 and outperforms SimPO and SPPO on Arena-Hard, MT-Bench, and WildBench. The work directly addresses the gap between verifiable-reward online RL (strong on math/code) and preference optimization (strong on open-ended tasks).
Gradient Equilibrium shown equivalent to Blackwell Approachability in online learning
A new arXiv preprint proves that gradient equilibrium (GEQ), a recently introduced online optimization framework generalizing first-order stationarity, is algorithmically equivalent to Blackwell approachability. The equivalence implies GEQ is also equivalent to regret minimization and calibration, resolving an open question about GEQ's place in the online learning landscape. The reductions are efficient and allow transfer of refined guarantees like optimism and strong adaptivity from regret minimization to GEQ, with applications including online conformal prediction.
Language models linearly encode a 'value axis' tracking expected goal success, study finds
Researchers construct a 'value axis' in Qwen3-8B's activation space using synthetic in-context RL data, finding that this axis distinguishes high vs. low confidence, backtracking vs. non-backtracking rollouts, and correct vs. corrupted code. Steering along this axis causally modulates self-correction behavior and verbosity, while DPO training shifts the internal value of rewarded behaviors. Applied to real-world settings, the axis reveals that Qwen assigns low internal value to politically sensitive queries post-training and that SFT increases domain-specific confidence. The findings suggest LLMs linearly encode an estimate of expected goal success that shapes their generative behavior.
Reward uncertainty as a principled mechanism for diverse RL behaviour
A new arXiv preprint proposes replacing the scalar reward in RL with a distribution over reward functions, applying a non-linear objective over sets of actions to induce calibrated behavioural diversity without sacrificing expected reward. The authors derive a principled gradient estimator in the contextual bandit setting and prove the formulation generalizes vanilla policy gradient and action-set approaches. The work is motivated by applications like language model fine-tuning where diversity is desirable but entropy regularization and diversity bonuses introduce fragile trade-offs. Empirical results support the framework as a theoretically grounded alternative to heuristic diversity methods.
Repeated Policy Regret (RP-Regret): Regret minimization against adaptive opponents in repeated games
This arXiv paper introduces Repeated Policy Regret (RP-Regret), a new game-theoretic metric for regret minimization in repeated games where opponents can adapt based on play history — a setting where standard external regret fails. The authors prove necessary conditions for sublinear RP-Regret and propose three algorithms to minimize it, including oracle-based, linearized surrogate, and slow-opponent variants. When all players minimize RP-Regret, certain subgame perfect equilibria can be learned, and experiments show more cooperative outcomes in games like Stag-Hunt.
RL without TD Learning: Divide-and-Conquer Value Learning for Long-Horizon Off-Policy RL
A BAIR blog post introduces a divide-and-conquer paradigm for off-policy reinforcement learning that avoids temporal difference (TD) learning's error accumulation problem by reducing Bellman recursions logarithmically rather than linearly. The approach leverages the triangle inequality structure of goal-conditioned RL to define a transitive Bellman update rule, enabling value learning that scales to long-horizon tasks. The authors claim this is the first practical realization of divide-and-conquer value learning at scale in goal-conditioned RL settings, building on an idea traceable to Kaelbling (1993). The post frames this as a third paradigm alongside TD and Monte Carlo methods, addressing a key gap in scalable off-policy RL.
Scaling Laws for Reward Model Overoptimization
OpenAI published research investigating how reward model overoptimization scales with policy and reward model size in RLHF pipelines. The work characterizes the relationship between KL divergence from the initial policy and gold-standard reward, finding predictable degradation patterns as optimization pressure increases. This provides empirical grounding for understanding Goodhart's Law dynamics in language model fine-tuning and has implications for designing safer, more robust RLHF training regimes.