4arXiv cs.LG (Machine Learning)·
Reinformed Dreamer: Latent guidance improves asymmetric world model learning in MBRL
A new arXiv preprint introduces Reinformed Dreamer, a model-based reinforcement learning algorithm that extends the Informed Dreamer approach with a novel asymmetric representation learning objective using latent guidance. The authors identify a limitation in how Informed Dreamer learns privileged information representations and propose a fix that yields more consistent benchmark improvements. The work contributes to the asymmetric RL paradigm, which leverages additional state information during training beyond reward signals.
Related guides (1)
Related events (8)
Mask-Aware Policy Gradients improve RL training for Masked Diffusion Language Models
A new arXiv preprint introduces a two-stage action MDP formalization for applying reinforcement learning to Masked Diffusion Language Models (MDLMs), decomposing the policy gradient into a token prediction term and a masking order term. Prior approaches ignored the position-unmasking decision, leading to intractable log-likelihood estimates; the proposed method optimizes both terms jointly. The approach achieves 87.1% on GSM8K and 53.4% on MBPP, claiming state-of-the-art results for MDLM-based reasoning and coding.
Sleep paradigm for LLMs enables continual learning and memory consolidation via distillation and RL
A new arXiv preprint proposes a 'Sleep' paradigm for language models that enables continual learning by consolidating short-term in-context memories into long-term parameters. The framework has two stages: Knowledge Seeding (distilling a smaller model's memories into a larger network via on-policy distillation combined with RL-based imitation learning) and Dreaming (self-improvement via RL-generated synthetic curricula without human supervision). Experiments cover long-horizon tasks, continual learning, knowledge incorporation, and few-shot generalization, addressing a known weakness of current LLMs in retaining temporal knowledge across contexts.
Reinforcement Learning with Metacognitive Feedback (RLMF) improves LLM calibration and uncertainty expression
Researchers introduce Reinforcement Learning with Metacognitive Feedback (RLMF), a training paradigm that refines preference optimization using a model's self-judgments of its own performance quality. The method is applied to faithful calibration — aligning a model's expressed confidence with its intrinsic uncertainty — and achieves state-of-the-art results across diverse tasks while outperforming standard RL by up to 63%. A companion technique, metacognitive data selection, uses similar self-judgments to identify high-value training examples, outperforming naive active learning baselines. The work positions metacognitive performance as a novel and effective RL signal for improving LLM reliability and alignment.
DREAM: Training dense retrieval embeddings via autoregressive LLM supervision
DREAM is a new method for training dense retrieval embedding models using the autoregressive next-token prediction objective of a frozen LLM, bypassing the need for labeled positive/negative document pairs required by contrastive training. The approach injects retriever-generated query-document similarity scores into selected attention heads of the LLM, allowing prediction loss gradients to flow back to the retriever. Evaluated on BEIR and RTEB benchmarks with 0.5B–3B parameter backbones, DREAM consistently outperforms contrastive baselines across model scales.
Agency-transferring technique improves RL policy training by bootstrapping from baseline policies
A new arXiv paper proposes a model-free reinforcement learning method that embeds an existing suboptimal baseline policy into training via an arbitration mechanism, progressively transferring control from the baseline to a trainable neural network. The approach yields high goal-reaching rates from the start of training and produces a standalone policy that outperforms the baseline without requiring it at inference time. Theoretical bounds on goal-reaching probability are derived, and empirical results on continuous-control benchmarks show competitive or superior returns compared to existing methods.
KGRL: Neuro-symbolic reinforcement learning for parametrized action spaces using Datalog knowledge bases
A new arXiv preprint introduces KGRL (Knowledge- and Gradient-Guided Reinforcement Learning), a neuro-symbolic algorithm for Parametrized Action Markov Decision Processes (PAMDPs) where each decision involves both a symbolic action and continuous numerical parameters. KGRL uses a Datalog knowledge base to prune infeasible actions and constrain parameter spaces, then applies gradient-based refinement to estimate optimal parameters during training and deployment. The approach improves sample efficiency and episodic return over state-of-the-art PAMDP baselines, and provides local procedural explanations for its decisions.
Progress Advantage: Annotation-Free Step-Level Scoring for LLM Agents via RL Post-Training
Researchers introduce 'progress advantage,' a method that derives implicit step-level reward signals for LLM agents directly from the log-probability ratio between an RL-trained policy and its reference policy, without requiring dedicated process reward model training. The approach is shown to recover the optimal advantage function under a general stochastic MDP formulation, making it annotation-free and domain-agnostic. Validated across five benchmarks and four model families on tasks including test-time scaling, uncertainty quantification, and failure attribution, it outperforms confidence-based baselines and even dedicated trained reward models. The result is practically significant because building process reward models for agentic settings is currently a major bottleneck.
VRRL: Reinforcement learning framework for visually grounded self-reflection in vision-language models
Researchers propose VRRL, a reinforcement learning training framework designed to improve self-reflection in large vision-language models (LVLMs) by ensuring corrections are grounded in visual inputs rather than purely textual reasoning. The framework uses two mechanisms: random masking of trajectory prefixes to emphasize error recovery, and buffered roll-ins from an experience replay buffer to expose models to diverse failure states. Evaluated on visual grounding tasks (tables, charts) and spatial navigation benchmarks, VRRL substantially improves out-of-distribution accuracy over standard RL and reflection-oriented fine-tuning baselines. The work addresses a concrete failure mode where LVLMs lose visual grounding during chain-of-thought reflection.