5arXiv cs.CL (Computation and Language)·
RL-trained LLMs show reduced task conflicts and better model merging than SFT-trained models
A new arXiv preprint systematically compares model merging behavior between RL-trained and SFT-trained LLMs across five tasks, finding that RL significantly reduces task conflicts and performance degradation post-merge. The authors identify three mechanisms: smaller gradient updates from on-policy training, an optimization objective that limits conflict parameter updates at convergence, and joint positive/negative example optimization that steers models toward unbiased task-specific parameter subspaces. The findings suggest RL training paradigms are inherently better suited for model merging workflows, with implications for how practitioners choose training methods when multi-task consolidation is a goal.
Related guides (2)
Related events (8)
Model merging matches joint multi-task RL training on AppWorld benchmark, explained by near-orthogonal task vectors
A new arXiv paper provides the first direct comparison of model merging versus joint multi-task reinforcement learning training, using Qwen3-8B specialists trained on the AppWorld agent benchmark with the LOOP algorithm. Merging methods (TIES, RAM+) statistically match jointly trained models on task-goal completion. The authors explain this via task vector geometry: specialist task vectors are near-orthogonal (cosine similarity 0.06–0.10) despite ~65% parameter support overlap, causing sign- and support-based merging methods to collapse to near-uniform averaging.
Instruction tuning plus model merging improves reasoning LMs in verifiable and unverifiable domains
A new arXiv preprint proposes a two-step method for adapting reasoning language models (RLMs) to domains lacking reliable verifiers: first apply standard supervised fine-tuning (without reasoning traces) on the target domain, then merge the resulting model with the original RLM to recover reasoning behavior. The technique is evaluated on both verifiable domains (coding) and hard-to-verify domains (text summarization), showing improvements while preserving general RLM capabilities. Notably, the authors report the full adaptation can be achieved for under $3, making it highly accessible.
Controlled study reveals how pretraining shapes RL post-training returns for LLM reasoning
A new arXiv preprint uses chess as a controlled testbed to study the relationship between pretraining and RL post-training across the full LLM training pipeline, spanning models from 5M to 1B parameters. Key findings: post-RL performance is well-predicted by pretraining loss, RL reward curve slope improves approximately linearly with pretraining tokens, and RL behaves differently on easy versus hard problems (amplifying already-preferred moves vs. surfacing near-absent correct moves). The predictive pattern transfers to a math-domain 1B model, suggesting the findings generalize beyond the chess testbed.
Reinforcement Learning with Metacognitive Feedback (RLMF) improves LLM calibration and uncertainty expression
Researchers introduce Reinforcement Learning with Metacognitive Feedback (RLMF), a training paradigm that refines preference optimization using a model's self-judgments of its own performance quality. The method is applied to faithful calibration — aligning a model's expressed confidence with its intrinsic uncertainty — and achieves state-of-the-art results across diverse tasks while outperforming standard RL by up to 63%. A companion technique, metacognitive data selection, uses similar self-judgments to identify high-value training examples, outperforming naive active learning baselines. The work positions metacognitive performance as a novel and effective RL signal for improving LLM reliability and alignment.
Make LLM Fine-tuning 2x faster with Unsloth and 🤗 TRL
Hugging Face published a blog post detailing an integration between Unsloth and TRL (Transformer Reinforcement Learning) library that claims to achieve 2x faster LLM fine-tuning. The post covers how Unsloth optimizes training kernels to reduce memory usage and increase throughput. This is relevant to practitioners looking to reduce compute costs and time for fine-tuning large language models.
Paper diagnoses RL collapse in multi-step tool-use training and proposes supervisory signal fixes
A new arXiv preprint identifies a failure mode in reinforcement learning for LLM tool use: catastrophic collapse caused by probability spikes in control tokens that disrupt structured execution while leaving underlying tool-use capability intact. The authors systematically evaluate supervisory signals—including off-policy supervision, hint-based guidance, and erroneous example supervision—under synchronous and interleaved training schemes. Interleaving SFT with RL improves stability but degrades performance under out-of-distribution format and content evaluation. Code is released as Tool-RL-Box.
RL-trained LLMs learn retriever-specific query formulation strategies for RAG
A new arXiv paper presents the first systematic study of using reinforcement learning to teach LLMs to adapt query formulation strategies to different retrieval backends. The authors find that different retrievers have surprisingly distinct optimal query styles (e.g., descriptive vs. question-like), making cross-retriever strategy transfer ineffective. They introduce a branching-based rollout technique to stabilize training over multi-step retrieval trajectories and show gains from retriever-specific human guidance and model scaling.
MOPD: Multi-Teacher On-Policy Distillation for integrating multiple RL-trained capabilities in LLMs
Researchers propose Multi-teacher On-Policy Distillation (MOPD), a post-training paradigm that first trains domain-specialized RL teacher models, then distills them into a student model using on-policy rollouts to eliminate exposure bias. Evaluated on Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines while preserving nearly all per-domain capability. The method has been deployed in production for MiMo-V2-Flash, an industrial-scale frontier model, validating its practical applicability. The approach also enables parallel, decoupled development of domain teachers, reducing cross-domain interference in multi-capability post-training.