7arXiv cs.CL (Computation and Language)·
Agon: Competitive cross-model RL uses rival models as implicit reasoning graders
Agon is a new reinforcement learning framework where two competing models grade each other implicitly by attempting the same problems in alternating roles — one drafts a solution, the other reads it while solving, and each is rewarded for out-solving the rival. This sidesteps the need for process labels or a reward model, and because both models are jointly optimized, each faces a progressively stronger opponent. On the hard split of DeepMath with Qwen3, Agon doubles GRPO's pass@1, roughly eight times the gain of an untrained Mixture-of-Agents baseline, with results replicating on competitive programming and across model families.
Related guides (3)
Related events (8)
AdvGRPO: Stable co-training framework for adaptive red teaming of language models
Researchers introduce AdvGRPO, a co-training framework that makes GRPO viable for joint attacker-defender optimization in LLM red teaming, addressing previously reported instability. The method uses dense multi-channel rewards and decoupled advantage normalization, with a curriculum progressing from single-turn to multi-turn attacks before bootstrapping co-training. Co-trained defenders outperform baselines on safety benchmarks, and the attacks show transferability across models.
Agora: Auction-based task allocation framework for LLM agent reasoning
Researchers introduce Agora, a framework that uses an incentive-compatible auction mechanism to dynamically route reasoning subtasks to the most capable expert models or tools, rather than relying on coarse-grained function matching. Agents bid based on 'rectified competence' to prevent overconfident solvers from capturing critical logic steps. Evaluations across five benchmarks show improvements over single-model, routing, and cascade baselines, with a controllable cost-quality trade-off via a single auction parameter.
Learning to Cooperate, Compete, and Communicate
OpenAI published early research on multiagent environments as a pathway toward AGI, arguing that competitive multi-agent settings provide a natural curriculum and continuous pressure for improvement. The post highlights two key properties: difficulty scales with competitor skill, and no stable equilibrium exists, ensuring perpetual learning pressure. The work positions multiagent environments as fundamentally different from single-agent RL and calls for significant further research.
N-GRPO: Semantic Neighbor Mixing for Improved Policy Optimization in LLM Reasoning
A new arXiv preprint introduces N-GRPO, an exploration strategy for the GRPO reinforcement learning framework that improves solution diversity during rollout by mixing embeddings of anchor tokens with their nearest semantic neighbors rather than using token-level sampling or random noise. The method is evaluated on DeepSeek-R1-Distill-Qwen models of various sizes and shows consistent improvements on math reasoning benchmarks plus out-of-distribution generalization. The work targets a known limitation in RLHF-style training: redundant rollout trajectories that reduce effective learning signal.
OPERA: Perplexity-based RL alignment for open-ended reasoning tasks
OPERA (Objective Perplexity-based Reflective Alignment) proposes replacing LLM-as-a-judge reward models with intrinsic rewards derived from perplexity dynamics to stabilize RL training on open-ended tasks like creative writing. The method includes a cold-start data synthesis pipeline generating 20,000 reasoning trajectories using perplexity-prioritized rollouts. Applied to Qwen3-8B, OPERA claims state-of-the-art among open-source models on open-ended tasks, reportedly matching or exceeding Gemini 2.5 and MiniMax-M2.5 on some benchmarks.
AXPO: Agent Explorative Policy Optimization Addresses Thinking-Acting Gap in Multimodal Agentic Reasoning
This paper identifies a structural asymmetry in agentic reasoning called the 'Thinking-Acting Gap,' where tool use is attempted in only ~30% of rollouts under standard RL training (GRPO), and all-wrong tool-using subgroups suppress learning signals. The authors propose AXPO (Agent eXplorative Policy Optimization), which fixes the thinking prefix and resamples tool calls for all-wrong subgroups, combined with uncertainty-based prefix selection. Evaluated across nine multimodal benchmarks on Qwen3-VL-Thinking at multiple scales, SFT+AXPO outperforms SFT+GRPO by +1.8pp on both Pass@1 and Pass@4 at 8B, with the 8B model surpassing the 32B baseline on Pass@4 using 4× fewer parameters.
GGRO: Gradient-Guided Reward Optimization for inference-time LLM alignment
Researchers introduce Gradient-Guided Reward Optimization (GGRO), an inference-time alignment method that uses gradient signals from a reward model to inject 'nudging tokens' at high-uncertainty decoding steps, rather than relying on sampling-intensive re-ranking approaches like Best-of-N. The method monitors token-level entropy to detect distribution drift and steers generation trajectories directly, claiming improved robustness to reward hacking with minimal computational overhead. Experiments show gains across safety, helpfulness, and reasoning benchmarks compared to standard inference-time alignment baselines.
RiVER framework enables RL training of LLMs on tasks without ground-truth solutions
Researchers introduce RiVER (Ranking-induced VERifiable framework), a reinforcement learning approach that trains LLMs on score-based optimization tasks using deterministic execution feedback as continuous rewards, without requiring ground-truth answers. The method addresses two failure modes in group-relative RL with continuous rewards—scale dominance and frequency dominance—via calibrated, instance-wise reward shaping. Applied to Qwen3-8B and GLM-Z1-9B-0414 on competitive programming tasks, RiVER improves ALE rating rank by ~9% and also transfers to exact-solution benchmarks (LiveCodeBench, USACO) with 2-4% absolute gains, unlike raw-score baselines. The result suggests score-based heuristic tasks can serve as general-purpose RL training environments for coding ability.