5arXiv cs.CL (Computation and Language)·
GRPO outperforms SFT for ASR adaptation using only synthetic speech
A new arXiv preprint demonstrates that Group Relative Policy Optimization (GRPO) substantially outperforms supervised fine-tuning (SFT) when adapting LLM-based automatic speech recognition to regulated domains using only synthetic TTS data. GRPO alone reduces word error rate by 40% relative to SFT (36.71% → 22.09%), with an SFT+GRPO combination achieving 45% relative reduction. The authors attribute gains to behavioral changes — improved stopping calibration and better audio-text alignment — rather than representational shifts in early layers.
Related guides (3)
Related events (8)
P4IR framework uses SFT + GRPO to improve LLM-based automated building code compliance
Researchers introduce P4IR, a two-stage framework combining supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) to improve LLM accuracy in automated code compliance (ACC) for building regulations. The approach reduces tree edit distance and token-level Levenshtein distance by up to 23.8% and 38.6% respectively versus SFT baselines, and outperforms Claude Opus/Sonnet 4.5, GPT-5.2, Qwen-3-Max, and GLM-4.7 in zero-shot settings. The work targets a narrow but practically important domain where LLM hallucinations carry real regulatory consequences.
AdvGRPO: Stable co-training framework for adaptive red teaming of language models
Researchers introduce AdvGRPO, a co-training framework that makes GRPO viable for joint attacker-defender optimization in LLM red teaming, addressing previously reported instability. The method uses dense multi-channel rewards and decoupled advantage normalization, with a curriculum progressing from single-turn to multi-turn attacks before bootstrapping co-training. Co-trained defenders outperform baselines on safety benchmarks, and the attacks show transferability across models.
GSPO: Group Sequence Policy Optimization for Scalable RL Training of Language Models
Qwen researchers introduce Group Sequence Policy Optimization (GSPO), a new RL algorithm designed to address severe training instability and model collapse observed in existing methods like GRPO during extended training runs. The core motivation is enabling stable RL scaling for language models to improve reasoning and problem-solving capabilities with increased compute. The paper targets a known bottleneck in post-training pipelines where instability prevents further performance gains.
N-GRPO: Semantic Neighbor Mixing for Improved Policy Optimization in LLM Reasoning
A new arXiv preprint introduces N-GRPO, an exploration strategy for the GRPO reinforcement learning framework that improves solution diversity during rollout by mixing embeddings of anchor tokens with their nearest semantic neighbors rather than using token-level sampling or random noise. The method is evaluated on DeepSeek-R1-Distill-Qwen models of various sizes and shows consistent improvements on math reasoning benchmarks plus out-of-distribution generalization. The work targets a known limitation in RLHF-style training: redundant rollout trajectories that reduce effective learning signal.
AdaPrefix-GRPO: Adaptive prefix control doubles GRPO accuracy on hard math reasoning
A new arXiv preprint introduces AdaPrefix-GRPO, a method that addresses GRPO's failure to learn from problems where no rollout succeeds by prepending correct solution prefixes and dynamically adjusting prefix length to maintain ~50% success rate per problem throughout training. The prefix assistance is gradually withdrawn so the final model solves problems unaided. On hard math benchmarks, the method more than doubles GRPO accuracy for a 0.6B model (2.1x) and achieves 1.7x improvement on AIME, while halving trace length, with larger gains on smaller models. The implementation requires only data preparation changes and a loss mask, leaving the trainer unchanged.
AXPO: Agent Explorative Policy Optimization Addresses Thinking-Acting Gap in Multimodal Agentic Reasoning
This paper identifies a structural asymmetry in agentic reasoning called the 'Thinking-Acting Gap,' where tool use is attempted in only ~30% of rollouts under standard RL training (GRPO), and all-wrong tool-using subgroups suppress learning signals. The authors propose AXPO (Agent eXplorative Policy Optimization), which fixes the thinking prefix and resamples tool calls for all-wrong subgroups, combined with uncertainty-based prefix selection. Evaluated across nine multimodal benchmarks on Qwen3-VL-Thinking at multiple scales, SFT+AXPO outperforms SFT+GRPO by +1.8pp on both Pass@1 and Pass@4 at 8B, with the 8B model surpassing the 32B baseline on Pass@4 using 4× fewer parameters.
One-shot GRPO training on a single biased example can break LLM alignment
A new arXiv paper demonstrates that a single biased training example using Group Relative Policy Optimization (GRPO) is sufficient to induce systematic bias in aligned LLMs, with stereotype-driven reasoning generalizing across attributes, categories, and benchmarks. The authors find that model susceptibility varies based on the initial likelihood of producing biased outputs. The result exposes a critical vulnerability in post-training alignment: a minimal fine-tuning intervention can override safety guardrails.
Adaptive Data Scheduling (ADS) improves LLM reinforcement learning post-training by 5.2% over GRPO
Researchers propose Adaptive Data Scheduling (ADS), a dual-level framework that replaces uniform sampling in RL post-training with adaptive distribution over semantic clusters and policy-boundary sample selection. Evaluated across three LLMs and seven reasoning benchmarks, ADS improves average accuracy by 5.2% over GRPO and generalizes across RL objectives. The method addresses a structural limitation in standard RL post-training pipelines by accounting for semantic data structure and evolving policy capability during training.