6arXiv cs.CL (Computation and Language)·
RL post-training composes primitive skills into higher-level reasoning strategies, not just amplification
A new arXiv paper investigates whether RL post-training creates genuinely new compositional reasoning strategies or merely amplifies latent base-model skills. Using a controlled rewrite-grammar environment with auditable traces, the authors show that RL solves problems the pretrained model rarely solves even with large sampling budgets, and that it does so by reorganizing primitive competencies into sequential and parallel compositional procedures. The key differentiator from rejection fine-tuning is selectivity: RL concentrates exploration into valid, reusable structures rather than producing high-volume but shortcut-laden rewrites.
Related guides (2)
Related events (8)
Controlled study reveals how pretraining shapes RL post-training returns for LLM reasoning
A new arXiv preprint uses chess as a controlled testbed to study the relationship between pretraining and RL post-training across the full LLM training pipeline, spanning models from 5M to 1B parameters. Key findings: post-RL performance is well-predicted by pretraining loss, RL reward curve slope improves approximately linearly with pretraining tokens, and RL behaves differently on easy versus hard problems (amplifying already-preferred moves vs. surfacing near-absent correct moves). The predictive pattern transfers to a math-domain 1B model, suggesting the findings generalize beyond the chess testbed.
ExpRL: RL-based mid-training using human QA data as reward scaffolds for LLM reasoning
ExpRL proposes an automated approach to LLM mid-training that replaces manually curated reasoning traces with large corpora of human-written QA data used as reward scaffolds rather than imitation targets. Reference solutions are hidden from the policy and used only to construct problem-specific grading rubrics, enabling dense process-level rewards that reinforce partial progress and intermediate reasoning steps. On challenging math reasoning benchmarks, ExpRL outperforms SFT, sparse-reward GRPO, and self-distillation as an RL initialization strategy, with additional mixed-domain experiments suggesting broader applicability.
RA-RFT: Retrieval-Augmented Reinforcement Fine-Tuning teaches LLMs to reason by analogy
Researchers propose Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT), a post-training framework that trains a retriever to rank contexts by expected reasoning benefit rather than semantic similarity, then fine-tunes a policy model via reinforcement learning using retrieved analogous demonstrations. The key insight is that reasoning-relevant retrieval surfaces complementary solution strategies rather than superficially similar problems. On mathematical reasoning benchmarks, RA-RFT improves AIME 2025 average@32 accuracy by 7.1 and 2.8 points over GRPO for Qwen3-1.7B and Qwen3-4B respectively, suggesting reasoning-aware retrieval is orthogonal to reward design and training curriculum improvements.
Rubric-Conditioned Self-Distillation: structured feedback for reasoning model post-training
A new arXiv preprint proposes Rubric-Conditioned Self-Distillation (RCSD), a post-training framework that replaces scalar reward signals and noisy chain-of-thought annotations with structured rubrics for fine-grained credit assignment. The method conditions a teacher model on criterion-level rubrics to provide token-level guidance on the student's own sampled trajectories, avoiding reliance on a single reference rationale. Evaluated on science reasoning benchmarks, RCSD outperforms GRPO by 1.0 points and OPSD by 0.9 points on average.
Single transformer layer training can match full-parameter RL post-training in LLMs
A new arXiv paper challenges the assumption that all transformer layers contribute equally during RL post-training, finding that training a single layer can recover most or all of the gains from full-parameter RL. The authors introduce a 'layer contribution' metric and evaluate across seven models from the Qwen2.5 and Qwen3 families, three RL algorithms (GRPO, GiGPO, Dr. GRPO), and tasks including math reasoning, code, and agentic decision-making. A consistent structural pattern emerges: high-contribution layers concentrate in the middle of the transformer stack, and this ranking is stable across datasets, tasks, and algorithms.
RACES framework enables recursive composition of verifiable RL environments for LLM reasoning generalization
RACES (Recursive Automated Composition for Environment Scaling) is a new framework that treats verifiable RL training environments as composable building blocks, automatically fusing them when input/output types match. The system implements 300 base environments and four composition operators (SEQUENTIAL, PARALLEL, SORT, SELECT) to generate diverse reasoning patterns at scale. Experiments show consistent gains on unseen benchmarks: DeepSeek-R1-Distill-Qwen-14B improves from 48.2 to 51.3 and Qwen3-14B from 58.8 to 61.1 averaged across six benchmarks. Notably, RACES achieves parity with 300 individual environments using only 50 base environments, suggesting strong efficiency gains over linear environment scaling.
Instruction tuning plus model merging improves reasoning LMs in verifiable and unverifiable domains
A new arXiv preprint proposes a two-step method for adapting reasoning language models (RLMs) to domains lacking reliable verifiers: first apply standard supervised fine-tuning (without reasoning traces) on the target domain, then merge the resulting model with the original RLM to recover reasoning behavior. The technique is evaluated on both verifiable domains (coding) and hard-to-verify domains (text summarization), showing improvements while preserving general RLM capabilities. Notably, the authors report the full adaptation can be achieved for under $3, making it highly accessible.
RePro: Retrospective Progress-Aware Self-Refinement for LLM Agent Training
Researchers introduce RePro (Retrospective Progress-Aware Training), a framework addressing the gap between step-wise RL optimization and metacognitive task-progress awareness in LLM agents. The approach uses a forward-then-reflect rollout paradigm where agents execute actions online and then retrospectively assess step-wise progress given the completed trajectory and known outcome. Evaluated on WebShop, ALFWorld, and Sokoban, RePro achieves up to 12% absolute success rate gains over baseline Qwen-family models without requiring continuous external supervision.