6arXiv cs.AI (Artificial Intelligence)·
Reinforcement learning for code optimization via calibrated timing rewards and adapted GRPO
A new arXiv preprint addresses the challenge of extending RL-based code generation to code optimization, where execution time is added to the reward signal. The authors identify three failure modes (measurement noise, reward sparsity, GRPO instability) and propose a three-stage solution: a new benchmark (DMC-Optim) with calibrated timing, a composed correctness-plus-speed reward with offline simulator, and adapted GRPO training. On Qwen 2.5 7B and CWM 32B, strict top-50% pass@1 improves from 18.0% to 31.3% and 30.7% to 50.4% respectively, with larger gains at stricter percentiles, while preserving correctness scores.
Related guides (3)
Related events (8)
GGRO: Gradient-Guided Reward Optimization for inference-time LLM alignment
Researchers introduce Gradient-Guided Reward Optimization (GGRO), an inference-time alignment method that uses gradient signals from a reward model to inject 'nudging tokens' at high-uncertainty decoding steps, rather than relying on sampling-intensive re-ranking approaches like Best-of-N. The method monitors token-level entropy to detect distribution drift and steers generation trajectories directly, claiming improved robustness to reward hacking with minimal computational overhead. Experiments show gains across safety, helpfulness, and reasoning benchmarks compared to standard inference-time alignment baselines.
ACPO: Adaptive Clip Policy Optimization improves RLVR training for LLM reasoning
A new arXiv preprint provides theoretical analysis of Reinforcement Learning from Verifiable Rewards (RLVR) updates, identifying off-policy degree and gradient expectation as key factors governing update dynamics. The authors show that differences in gradient steps per rollout substantially affect importance sampling ratio distributions and which tokens dominate updates. Based on this analysis, they propose Adaptive Clip Policy Optimization (ACPO), which adjusts clipping boundaries per token group by empirical variance of importance sampling ratios, outperforming DAPO and CISPO baselines on 3B and 7B models across math, tabular QA, and logic benchmarks.
Extrapolative Weight Averaging Reveals Correctness-Efficiency Frontiers in Code RL
This paper investigates whether extrapolative weight averaging of RL-trained checkpoints can extend Pareto frontiers between competing objectives (correctness vs. computational efficiency) without additional training. Starting from a shared initialization, the authors train checkpoints under nested unit-test coverage regimes for competitive programming tasks, revealing a correctness-efficiency frontier where higher-coverage rewards reduce optimization failures but increase correctness failures. Extrapolation beyond trained endpoints produces complementary policies that, when ensembled, improve pass@250 on LCB/hard by 3.3% over the best single checkpoint at matched sample budget. Results hold across 7B and 32B model scales and three inference settings: pure reasoning, tool use, and agentic coding.
Off-Context GRPO uses privileged guidance to overcome zero-reward cliff in hard reasoning tasks
Researchers introduce Off-Context GRPO (OC-GRPO), a variant of the GRPO reinforcement learning algorithm that addresses the 'zero learning signal' failure mode when models cannot solve hard problems. The method uses privileged guidance (e.g., solution prefixes) during training rollouts while applying an importance-corrected objective to avoid distributional mismatch with the unguided target. On standard mathematical reasoning benchmarks, OC-GRPO achieves a 3.9% absolute (13.8% relative) improvement over vanilla GRPO with negligible added cost.
POW3R: Policy-Aware Rubric Rewards for More Efficient RLVR Training
This paper identifies a failure mode in rubric-based reinforcement learning with verifiable rewards (RLVR): static aggregation of criterion weights conflates human-assigned importance with current optimization utility, causing many criteria to be either already saturated or unreachable. The authors introduce POW3R, a framework that dynamically reweights criterion-level rewards during training using rollout-level contrast to emphasize criteria that currently differentiate policy outputs. Across three base policies and two datasets (multimodal and text-only), POW3R wins 24 of 30 comparisons on rubric reward and strict completion metrics, and reaches equivalent performance in 2.5–4× fewer training steps than vanilla GRPO with rubric rewards.
SCOPE-RL framework densifies reward signals in RLVR to improve reasoning accuracy and token efficiency
SCOPE-RL introduces a two-stage reinforcement learning framework that addresses sparse reward limitations in RLVR by adding prefix-decomposed sub-question rewards before success and correctness-gated process-shape rewards after success. Applied to Qwen3-8B-Instruct on math reasoning datasets, the method improves average accuracy by up to 11.2 percentage points and reduces reasoning tokens by up to 27.1% over outcome-only GRPO. The gains generalize across GSPO and a smaller Qwen3-0.6B model, suggesting reward-signal densification is broadly complementary to existing RLVR advances. Code and data are publicly released.
Continual-learning evaluation on Terminal-Bench 2.0 tests whether agent optimizer gains compound across tasks
A new arXiv paper introduces a two-phase continual-learning evaluation framework built on Terminal-Bench 2.0 to test whether agent-optimization gains persist and compound when new tasks arrive over time. Three agent-harness optimization methods — GEPA, Meta Harness, and RELAI-VCL — are compared under identical budgets; all improve in static single-phase settings but diverge sharply under continual optimization. RELAI-VCL is the only method that both transfers positively to unseen tasks and continues improving, reaching a 76.4% lifelong average pass rate versus 58.7% for the unoptimized baseline. The key finding is that compounding gains require regression control built into the optimization loop to prevent shortcut solutions.
AdaPrefix-GRPO: Adaptive prefix control doubles GRPO accuracy on hard math reasoning
A new arXiv preprint introduces AdaPrefix-GRPO, a method that addresses GRPO's failure to learn from problems where no rollout succeeds by prepending correct solution prefixes and dynamically adjusting prefix length to maintain ~50% success rate per problem throughout training. The prefix assistance is gradually withdrawn so the final model solves problems unaided. On hard math benchmarks, the method more than doubles GRPO accuracy for a 0.6B model (2.1x) and achieves 1.7x improvement on AIME, while halving trace length, with larger gains on smaller models. The implementation requires only data preparation changes and a loss mask, leaving the trainer unchanged.