5arXiv cs.CL (Computation and Language)·
SVR: Self-Verifying Refinement enables adaptive test-time compute via joint verdict-confidence RL
Researchers introduce Self-Verifying Refinement (SVR), a multi-turn reinforcement learning framework that trains language models to self-verify their own outputs and adaptively stop refinement without external verifiers. The model produces a solution, a correctness verdict, and a confidence score at each turn, continuing refinement only when uncertain; ground-truth labels are used only for reward construction, not at inference. Trained with GRPO on Qwen3.5-2B, SVR achieves competitive accuracy on seven math reasoning benchmarks using an average of only 2.99 inference turns, outperforming fixed-budget and oracle-guided baselines. The work advances the line of research on efficient, oracle-free test-time compute scaling.
Related guides (3)
Related events (8)
Self-Trained Verification (STV) Unlocks Training- and Test-Time Self-Improvement for Reasoning Models
This paper introduces Self-Trained Verification (STV), a method that trains a verifier to imitate a more informed version of itself by leveraging reference solutions as supervision signal, addressing the core bottleneck in both test-time verification-refinement loops and self-training pipelines. At test time, STV roughly doubles accuracy on hard math and achieves a 14x lift on scientific reasoning tasks. At training time, the authors combine STV with RL in a procedure called Verifier-in-the-Loop (ViL) training, yielding a 33% further gain in pass@1 over an already RL-converged generator, with standalone pass@1 climbing 30% relative past standard RL convergence. The work argues that verification quality, not generation, is the primary bottleneck for scaling reasoning on hard problems.
VRRL: Reinforcement learning framework for visually grounded self-reflection in vision-language models
Researchers propose VRRL, a reinforcement learning training framework designed to improve self-reflection in large vision-language models (LVLMs) by ensuring corrections are grounded in visual inputs rather than purely textual reasoning. The framework uses two mechanisms: random masking of trajectory prefixes to emphasize error recovery, and buffered roll-ins from an experience replay buffer to expose models to diverse failure states. Evaluated on visual grounding tasks (tables, charts) and spatial navigation benchmarks, VRRL substantially improves out-of-distribution accuracy over standard RL and reflection-oriented fine-tuning baselines. The work addresses a concrete failure mode where LVLMs lose visual grounding during chain-of-thought reflection.
Reinforcement learning with verifiable rewards improves small-model legal machine translation
Researchers evaluate multiple training paradigms for legal machine translation, comparing supervised fine-tuning and reinforcement learning with verifiable rewards (RLVR) on small models (Qwen3.5 4B/9B, Gemma 3 12B) against frontier reasoning models. Using the Swiss multilingual legal system as a testbed, they find RLVR outperforms SFT for legal NMT and brings small models close to frontier performance, though a gap remains. The study also observes diminishing returns from re-training as model size increases. Code and models are publicly released.
RiVER framework enables RL training of LLMs on tasks without ground-truth solutions
Researchers introduce RiVER (Ranking-induced VERifiable framework), a reinforcement learning approach that trains LLMs on score-based optimization tasks using deterministic execution feedback as continuous rewards, without requiring ground-truth answers. The method addresses two failure modes in group-relative RL with continuous rewards—scale dominance and frequency dominance—via calibrated, instance-wise reward shaping. Applied to Qwen3-8B and GLM-Z1-9B-0414 on competitive programming tasks, RiVER improves ALE rating rank by ~9% and also transfers to exact-solution benchmarks (LiveCodeBench, USACO) with 2-4% absolute gains, unlike raw-score baselines. The result suggests score-based heuristic tasks can serve as general-purpose RL training environments for coding ability.
LLM-as-a-Verifier: Training-free verification framework scales along granularity, repetition, and criteria decomposition
Researchers introduce LLM-as-a-Verifier, a general-purpose verification framework that treats verification as a new scaling axis for LLMs, computing continuous scores from token logit distributions rather than discrete judge outputs. The framework scales along three dimensions—score granularity, repeated evaluation, and criteria decomposition—and achieves state-of-the-art results on Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%) without requiring additional training. The authors also demonstrate that the framework's fine-grained signals can serve as dense RL feedback, improving sample efficiency for SAC and GRPO on robotics and math benchmarks, and build a Claude Code extension for monitoring agentic systems.
ExpRL: RL-based mid-training using human QA data as reward scaffolds for LLM reasoning
ExpRL proposes an automated approach to LLM mid-training that replaces manually curated reasoning traces with large corpora of human-written QA data used as reward scaffolds rather than imitation targets. Reference solutions are hidden from the policy and used only to construct problem-specific grading rubrics, enabling dense process-level rewards that reinforce partial progress and intermediate reasoning steps. On challenging math reasoning benchmarks, ExpRL outperforms SFT, sparse-reward GRPO, and self-distillation as an RL initialization strategy, with additional mixed-domain experiments suggesting broader applicability.
VERITAS: Visual verification enables inference-time steering and autonomous improvement for robot policies
Researchers introduce VERITAS, a generator-verifier framework pairing a pre-trained generalist robot policy with a gradient-free visual verifier to steer actions at inference time without additional training. Verified rollouts are also used for offline self-improvement via fine-tuning, achieving performance gains comparable to expert demonstrations but without human intervention. The work demonstrates that inference-time verification is a scalable mechanism for autonomous policy improvement during deployment.
Self-improving VLMs can silently regress when verifier quality is task-mismatched
A new arXiv paper demonstrates that verifier-driven self-DPO, a common recipe for self-improving visual-language models, can silently degrade student model performance when the verifier's task-rubric accuracy is insufficient for the target task. Experiments on Qwen-3-VL-2B and Qwen-2.5-VL-3B across MathVista, MMMU, and BLINK show regressions of 3.4–10.9 percentage points below frozen baselines, with the counterintuitive finding that more accurate-but-still-wrong verifiers cause larger regressions than near-random ones. The authors provide a mechanistic explanation via a variance theorem for progress-gated replay and offer operational guidance: measure target-task rubric accuracy before running any verifier-driven loop and rank verifiers by task-specific quality rather than parameter count.