6arXiv cs.AI (Artificial Intelligence)·
GEAR: Reward shaping method reduces repetitive copying in long-context LLM reasoning
Researchers identify 'repetitive copying' as a pervasive failure mode in long-context LLMs, where models copy input text into reasoning traces instead of solving problems, worsening with context length. They propose GEAR (Grounding Evidence-Aware Reward), a reinforcement learning reward shaping method that adds a grounding reward for overlap with key evidence and a penalty for overlap with irrelevant distractor context. GEAR shows consistent improvements of up to +4.6 average points over standard accuracy-based RL across multiple model scales and benchmarks, with larger gains at longer contexts and reduced reasoning trace length. The work also introduces an automated pipeline for constructing evidence-annotated training data from arbitrary documents.
Related guides (3)
Related events (8)
ExpRL: RL-based mid-training using human QA data as reward scaffolds for LLM reasoning
ExpRL proposes an automated approach to LLM mid-training that replaces manually curated reasoning traces with large corpora of human-written QA data used as reward scaffolds rather than imitation targets. Reference solutions are hidden from the policy and used only to construct problem-specific grading rubrics, enabling dense process-level rewards that reinforce partial progress and intermediate reasoning steps. On challenging math reasoning benchmarks, ExpRL outperforms SFT, sparse-reward GRPO, and self-distillation as an RL initialization strategy, with additional mixed-domain experiments suggesting broader applicability.
ReContext: Training-free recursive evidence replay improves LLM long-context reasoning
Researchers introduce RECONTEXT, a training-free inference-time method for improving long-context reasoning in LLMs. The approach uses model-internal relevance signals to build a query-conditioned evidence pool that is replayed before final generation, without modifying the original context, external memory, or context pruning. Experiments across eight long-context datasets at 128K context length show consistent improvements on Qwen3-4B, Qwen3-8B, and Llama3-8B. The authors provide a theoretical grounding via associative memory theory, framing attention as cue-trace association and replay as trace reactivation.
GGRO: Gradient-Guided Reward Optimization for inference-time LLM alignment
Researchers introduce Gradient-Guided Reward Optimization (GGRO), an inference-time alignment method that uses gradient signals from a reward model to inject 'nudging tokens' at high-uncertainty decoding steps, rather than relying on sampling-intensive re-ranking approaches like Best-of-N. The method monitors token-level entropy to detect distribution drift and steers generation trajectories directly, claiming improved robustness to reward hacking with minimal computational overhead. Experiments show gains across safety, helpfulness, and reasoning benchmarks compared to standard inference-time alignment baselines.
Experiential Learning proposes LLM-as-a-Coach to replace scalar RL rewards with textual feedback distillation
Researchers introduce Experiential Learning (EL), a post-training method that replaces scalar RL rewards with rich textual feedback from an LLM-as-a-Coach, which distills per-response assessments into transferable experiential knowledge used to condition a teacher model and update the policy via context distillation. Evaluated across two policy families with both self-feedback and proprietary model feedback, EL consistently outperforms rubric-based RL on open-ended tasks and shows better out-of-distribution generalization while mitigating reward hacking. The work targets a known weakness of RLHF-style training on non-verifiable tasks where scalar rewards lose fine-grained preference signal.
RA-RFT: Retrieval-Augmented Reinforcement Fine-Tuning teaches LLMs to reason by analogy
Researchers propose Retrieval-Augmented Reinforcement Fine-Tuning (RA-RFT), a post-training framework that trains a retriever to rank contexts by expected reasoning benefit rather than semantic similarity, then fine-tunes a policy model via reinforcement learning using retrieved analogous demonstrations. The key insight is that reasoning-relevant retrieval surfaces complementary solution strategies rather than superficially similar problems. On mathematical reasoning benchmarks, RA-RFT improves AIME 2025 average@32 accuracy by 7.1 and 2.8 points over GRPO for Qwen3-1.7B and Qwen3-4B respectively, suggesting reasoning-aware retrieval is orthogonal to reward design and training curriculum improvements.
LongTraceRL: Reinforcement Learning for Long-Context Reasoning via Search Agent Trajectories and Rubric Rewards
LongTraceRL is a new RL training framework for improving long-context reasoning in LLMs, addressing limitations of existing RLVR methods. It constructs challenging training data using multi-hop questions from knowledge graph random walks and tiered distractors derived from search agent trajectories (high-confusability: read but uncited; low-confusability: seen but unopened). A rubric reward provides entity-level process supervision along reasoning chains, applied only to correct responses to prevent reward hacking. Experiments across three LLMs (4B–30B parameters) on five long-context benchmarks show consistent improvements over strong baselines.
GR2: Generative Reasoning Re-Ranker applies RL with verifiable rewards to industrial-scale LLM recommendation
GR2 (Generative Reasoning Re-Ranker) is a new framework that applies reinforcement learning with verifiable rewards to the re-ranking stage of industrial recommendation systems, a step largely overlooked by prior LLM-based recommendation research. The system combines semantic ID mid-training, reasoning-trace distillation from a stronger teacher model, and purpose-built RL rewards, plus a context compressor and On-Policy Distillation to make it viable at scale. Deployed on industrial traffic, GR2 achieves +18.7% R@1 and +9.6% N@3 over legacy baselines. The paper also identifies a critical reward-hacking failure mode where LLMs exploit position bias or preserve input order, motivating conditional verifiable rewards.
STARE: Token-level advantage reweighting to prevent entropy collapse in GRPO-style RL training
Researchers introduce STARE, a method addressing policy entropy collapse in GRPO-style reinforcement learning from verifiable rewards (RLVR) for LLM post-training. Through first-order gradient analysis, they identify a token-level credit assignment mismatch and propose selectively reweighting advantages for entropy-critical tokens using batch-internal surprisal quantiles plus a closed-loop entropy gate. Evaluated across 1.5B–32B models on short/long chain-of-thought and multi-turn tool use tasks, STARE outperforms DAPO and other baselines by 4–8% on AIME24/25 while sustaining stable training over thousands of steps.