6arXiv cs.CL (Computation and Language)·
Controlled study reveals how pretraining shapes RL post-training returns for LLM reasoning
A new arXiv preprint uses chess as a controlled testbed to study the relationship between pretraining and RL post-training across the full LLM training pipeline, spanning models from 5M to 1B parameters. Key findings: post-RL performance is well-predicted by pretraining loss, RL reward curve slope improves approximately linearly with pretraining tokens, and RL behaves differently on easy versus hard problems (amplifying already-preferred moves vs. surfacing near-absent correct moves). The predictive pattern transfers to a math-domain 1B model, suggesting the findings generalize beyond the chess testbed.
Related guides (2)
Related events (8)
ExpRL: RL-based mid-training using human QA data as reward scaffolds for LLM reasoning
ExpRL proposes an automated approach to LLM mid-training that replaces manually curated reasoning traces with large corpora of human-written QA data used as reward scaffolds rather than imitation targets. Reference solutions are hidden from the policy and used only to construct problem-specific grading rubrics, enabling dense process-level rewards that reinforce partial progress and intermediate reasoning steps. On challenging math reasoning benchmarks, ExpRL outperforms SFT, sparse-reward GRPO, and self-distillation as an RL initialization strategy, with additional mixed-domain experiments suggesting broader applicability.
RL post-training composes primitive skills into higher-level reasoning strategies, not just amplification
A new arXiv paper investigates whether RL post-training creates genuinely new compositional reasoning strategies or merely amplifies latent base-model skills. Using a controlled rewrite-grammar environment with auditable traces, the authors show that RL solves problems the pretrained model rarely solves even with large sampling budgets, and that it does so by reorganizing primitive competencies into sequential and parallel compositional procedures. The key differentiator from rejection fine-tuning is selectivity: RL concentrates exploration into valid, reusable structures rather than producing high-volume but shortcut-laden rewrites.
Single transformer layer training can match full-parameter RL post-training in LLMs
A new arXiv paper challenges the assumption that all transformer layers contribute equally during RL post-training, finding that training a single layer can recover most or all of the gains from full-parameter RL. The authors introduce a 'layer contribution' metric and evaluate across seven models from the Qwen2.5 and Qwen3 families, three RL algorithms (GRPO, GiGPO, Dr. GRPO), and tasks including math reasoning, code, and agentic decision-making. A consistent structural pattern emerges: high-contribution layers concentrate in the middle of the transformer stack, and this ranking is stable across datasets, tasks, and algorithms.
ACPO: Adaptive Clip Policy Optimization improves RLVR training for LLM reasoning
A new arXiv preprint provides theoretical analysis of Reinforcement Learning from Verifiable Rewards (RLVR) updates, identifying off-policy degree and gradient expectation as key factors governing update dynamics. The authors show that differences in gradient steps per rollout substantially affect importance sampling ratio distributions and which tokens dominate updates. Based on this analysis, they propose Adaptive Clip Policy Optimization (ACPO), which adjusts clipping boundaries per token group by empirical variance of importance sampling ratios, outperforming DAPO and CISPO baselines on 3B and 7B models across math, tabular QA, and logic benchmarks.
Progress Advantage: Annotation-Free Step-Level Scoring for LLM Agents via RL Post-Training
Researchers introduce 'progress advantage,' a method that derives implicit step-level reward signals for LLM agents directly from the log-probability ratio between an RL-trained policy and its reference policy, without requiring dedicated process reward model training. The approach is shown to recover the optimal advantage function under a general stochastic MDP formulation, making it annotation-free and domain-agnostic. Validated across five benchmarks and four model families on tasks including test-time scaling, uncertainty quantification, and failure attribution, it outperforms confidence-based baselines and even dedicated trained reward models. The result is practically significant because building process reward models for agentic settings is currently a major bottleneck.
RL-trained LLMs learn retriever-specific query formulation strategies for RAG
A new arXiv paper presents the first systematic study of using reinforcement learning to teach LLMs to adapt query formulation strategies to different retrieval backends. The authors find that different retrievers have surprisingly distinct optimal query styles (e.g., descriptive vs. question-like), making cross-retriever strategy transfer ineffective. They introduce a branching-based rollout technique to stabilize training over multi-step retrieval trajectories and show gains from retriever-specific human guidance and model scaling.
Experiential Learning proposes LLM-as-a-Coach to replace scalar RL rewards with textual feedback distillation
Researchers introduce Experiential Learning (EL), a post-training method that replaces scalar RL rewards with rich textual feedback from an LLM-as-a-Coach, which distills per-response assessments into transferable experiential knowledge used to condition a teacher model and update the policy via context distillation. Evaluated across two policy families with both self-feedback and proprietary model feedback, EL consistently outperforms rubric-based RL on open-ended tasks and shows better out-of-distribution generalization while mitigating reward hacking. The work targets a known weakness of RLHF-style training on non-verifiable tasks where scalar rewards lose fine-grained preference signal.
LongTraceRL: Reinforcement Learning for Long-Context Reasoning via Search Agent Trajectories and Rubric Rewards
LongTraceRL is a new RL training framework for improving long-context reasoning in LLMs, addressing limitations of existing RLVR methods. It constructs challenging training data using multi-hop questions from knowledge graph random walks and tiered distractors derived from search agent trajectories (high-confusability: read but uncited; low-confusability: seen but unopened). A rubric reward provides entity-level process supervision along reasoning chains, applied only to correct responses to prevent reward hacking. Experiments across three LLMs (4B–30B parameters) on five long-context benchmarks show consistent improvements over strong baselines.