5arXiv cs.LG (Machine Learning)·
MIRROR: RL method improves VLM reasoning consistency across text, diagram, and combined views
Researchers introduce MIRROR (Modality-Informed Reciprocal Reasoning Optimization), a reinforcement learning approach that improves vision-language model reasoning by exploiting inconsistencies across different problem representations. The method constructs ODA-Data, a paired multimodal geometry dataset with text-dominant, image-dominant, and combined views, then uses the best-performing view as a teacher to train weaker views via reverse-KL divergence. MIRROR outperforms standard RL baselines on geometry reasoning benchmarks and produces more consistent cross-modal behavior. The work addresses a known gap between LLM and VLM reasoning capabilities on problems that have equivalent multi-modal representations.
Related guides (3)
Related events (8)
VRRL: Reinforcement learning framework for visually grounded self-reflection in vision-language models
Researchers propose VRRL, a reinforcement learning training framework designed to improve self-reflection in large vision-language models (LVLMs) by ensuring corrections are grounded in visual inputs rather than purely textual reasoning. The framework uses two mechanisms: random masking of trajectory prefixes to emphasize error recovery, and buffered roll-ins from an experience replay buffer to expose models to diverse failure states. Evaluated on visual grounding tasks (tables, charts) and spatial navigation benchmarks, VRRL substantially improves out-of-distribution accuracy over standard RL and reflection-oriented fine-tuning baselines. The work addresses a concrete failure mode where LVLMs lose visual grounding during chain-of-thought reflection.
AIR: Adaptive Interleaved Reasoning with Code in Multimodal LLMs via Reinforcement Learning
Researchers propose AIR, a system that trains multimodal large language models to adaptively interleave reasoning with code execution for numerical computation tasks, going beyond prior work that focused only on visual operations. The approach combines a two-stage cold-start data pipeline, RL dataset filtering, and a group-constrained reward function for tool-invocation decisions. Experiments show a 6.1 percentage point average improvement on evaluation benchmarks, with interleaved reasoning samples gaining 9.9 pp and tool-use success exceeding 95%.
CORA: Consistency-Oriented Reasoning Alignment addresses thinking-answer gap in multimodal RLVR
Researchers identify and analyze a systematic inconsistency between reasoning traces and final answers in RLVR-trained large vision-language models, showing the problem persists throughout GRPO training and inference. They propose CORA, which introduces a lightweight plug-and-play consistency reward model and a Hybrid Reward Advantage Splitting (HRAS) mechanism to coordinate task and consistency optimization. Experiments across multimodal reasoning benchmarks show CORA improves both task performance and reasoning faithfulness.
VAORA: Visual Action Outcome Reasoning Alignment for physical reasoning generalization in VLMs
Researchers introduce VAORA (Visual Action Outcome Reasoning Alignment), a reward design framework targeting two failure modes in vision-language models applied to interactive physical reasoning: hallucinated chain-of-thought reasoning and misalignment between reasoning and actions. VAORA uses two complementary rewards — a Visual Alignment Reward anchoring reasoning to visual context, and a Visual-Action Alignment Reward grounding reasoning in action outcomes — plus dense rewards from a pre-trained expert agent for training stability. Experiments on the PHYRE and Virtual Tool benchmarks show improved generalization to novel tasks and unseen environments.
ExpRL: RL-based mid-training using human QA data as reward scaffolds for LLM reasoning
ExpRL proposes an automated approach to LLM mid-training that replaces manually curated reasoning traces with large corpora of human-written QA data used as reward scaffolds rather than imitation targets. Reference solutions are hidden from the policy and used only to construct problem-specific grading rubrics, enabling dense process-level rewards that reinforce partial progress and intermediate reasoning steps. On challenging math reasoning benchmarks, ExpRL outperforms SFT, sparse-reward GRPO, and self-distillation as an RL initialization strategy, with additional mixed-domain experiments suggesting broader applicability.
ContextRL: Context-aware reinforcement learning improves grounding in agentic and multimodal LLMs
Researchers introduce ContextRL, a reinforcement learning method that trains LLMs to select the context that supports a given query-answer pair from two highly similar candidates, rather than supervising only final answers. The approach constructs contrastive context pairs in two domains: coding agent trajectories (1k pairs) and multimodal image pairs (7k pairs). ContextRL achieves +2.2% average gains over standard GRPO on 5 long-horizon benchmarks and +1.8% across 12 visual QA benchmarks, with ablations showing the gains stem from the context-selection objective rather than the contrastive data alone.
ETCHR: Decoupled Image Editing for Visual Chain-of-Thought Reasoning in MLLMs
ETCHR introduces a question-conditioned, reasoning-aware image editing model that decouples visual transformation from downstream understanding in multimodal LLMs. It addresses two identified gaps—language-side (mapping abstract questions to visual edits) and generation-side (edit quality degrading with reasoning depth)—via a two-stage training recipe combining supervised fine-tuning on edit trajectories and VLM-derived reward signals. Because the editor is decoupled, it plugs into arbitrary MLLMs without retraining, yielding Pass@1 gains of roughly +4.6 to +5.5 points across five task families when paired with Qwen3-VL-8B, Gemini-3.1-Flash-Lite, and Kimi K2.5. The work advances the 'think with images' paradigm beyond fixed toolkits and unified multimodal approaches.
Instruction tuning plus model merging improves reasoning LMs in verifiable and unverifiable domains
A new arXiv preprint proposes a two-step method for adapting reasoning language models (RLMs) to domains lacking reliable verifiers: first apply standard supervised fine-tuning (without reasoning traces) on the target domain, then merge the resulting model with the original RLM to recover reasoning behavior. The technique is evaluated on both verifiable domains (coding) and hard-to-verify domains (text summarization), showing improvements while preserving general RLM capabilities. Notably, the authors report the full adaptation can be achieved for under $3, making it highly accessible.