What GRPO is
Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm for post-training large language models on verifiable reward signals. Its defining characteristic is how it estimates the advantage of a generated output — the signal that tells the optimizer whether a particular response was better or worse than expected. Where PPO maintains a separately trained critic network to predict expected returns, GRPO sidesteps that entirely: it samples a group of outputs for each prompt, computes a reward for each, and normalizes those rewards within the group to produce advantages. The group mean is the baseline; no value model is required.
This design choice has two practical consequences. First, it substantially reduces memory and compute overhead relative to actor-critic methods. Second, it makes GRPO straightforward to implement on top of any verifiable reward — a format checker, a unit-test runner, a math answer verifier — which is why it became the default algorithm for reinforcement learning from verifiable rewards (RLVR) and the reference baseline against which virtually every new post-training method in the field is now compared.
How it works
The training loop is:
1. Sample a group of G completions from the current policy for each prompt. 2. Score each completion with a reward function (outcome-based, rubric-based, or process-based). 3. Normalize within the group: subtract the group mean, divide by the group standard deviation. These are the advantages. 4. Update the policy with a clipped surrogate objective (similar to PPO's clip) weighted by those advantages. 5. Repeat, typically with a KL penalty against a frozen reference policy to prevent reward hacking.
Because the baseline is computed from the model's own rollouts rather than a learned value function, GRPO is fully on-policy and requires no separate critic training phase.
Ecosystem and tooling
Hugging Face's TRL v1.0 (March 2026) ships GRPO as a first-class post-training primitive alongside PPO and DPO, marking its transition from research prototype to production-standard open-source infrastructure. The Qwen model families (Qwen2.5, Qwen3) are the most common base models in the published GRPO literature, appearing across math, code, agentic, and multimodal experiments.
Documented failure modes
The research community has catalogued GRPO's weaknesses with enough precision to spawn a generation of targeted fixes:
Entropy collapse. GRPO-style RLVR tends to collapse the policy's output distribution prematurely, killing exploration. STARE addresses this by identifying entropy-critical tokens via batch-internal surprisal quantiles and applying a closed-loop entropy gate; it outperforms DAPO by 4–8% on AIME24/25 across 1.5B–32B models.
Sparse rewards in agentic settings. When the environment provides sparse or non-natural-language feedback — empty brackets from a knowledge-graph API, for instance — GRPO exhibits a "peak-then-collapse" pattern where tool-grounded answer rates rise then fall to zero within 50 training steps. The PROVE framework addresses this for multi-step tool use by constructing programmatic, multi-component reward functions over stateful MCP environments, yielding gains of up to +10.2 on BFCL Multi-Turn.
The Thinking-Acting Gap. In multimodal agentic settings, tool use is attempted in only ~30% of rollouts under standard GRPO, and all-wrong tool-using subgroups suppress learning signals. AXPO fixes this by holding the thinking prefix constant and resampling tool calls for those subgroups, gaining +1.8pp on Pass@1 and Pass@4 at 8B scale.
Credit misassignment in long trajectories. Outcome-only credit punishes useful exploration in failed rollouts and reinforces redundant actions in successful ones. TRIAGE augments GRPO with role-typed process rewards (decisive progress, useful exploration, no-progress infrastructure, regression), improving success rates and reducing environment-facing turns by 10–15% on ALFWorld, Search-QA, and WebShop.
Reward signal quality. Static rubric aggregation can saturate or be unreachable for many criteria simultaneously. POW3R dynamically reweights criterion-level rewards using rollout-level contrast, reaching equivalent performance in 2.5–4× fewer training steps than vanilla GRPO with rubric rewards.
Efficiency improvements
Two orthogonal findings reduce GRPO's compute cost without changing its core algorithm:
- QuasiMoTTo replaces independent (i.i.d.) rollout sampling with correlated quasi-Monte Carlo samples, achieving equivalent GRPO training performance with 50% fewer training steps — a drop-in replacement requiring no model changes.
- Single-layer training: a study across seven Qwen2.5/Qwen3 models and three RL algorithms (GRPO, GiGPO, Dr. GRPO) finds that training only the highest-contribution layer — consistently located in the middle of the transformer stack — recovers most or all of full-parameter RL gains. The ranking is stable across datasets, tasks, and algorithms.
Variants and challengers
The GRPO ecosystem has bifurcated into refinements (which keep the group-relative mechanism and fix specific failure modes) and challengers (which replace the mechanism entirely):
Refinements include STARE (entropy), TRIAGE (credit), AXPO (agentic exploration), IH-GRPO (tool-integrated reasoning coherence), SCOPE-RL (reward densification via prefix-decomposed sub-question rewards, +11.2pp accuracy and –27.1% reasoning tokens on Qwen3-8B), and LA-GRPO (sparse functional token stabilization in multimodal models).
Challengers include:
- VPO (Vector Policy Optimization): replaces scalar rewards with vector-valued rewards to train for output diversity, targeting inference-scaling search procedures. Designed as a near-drop-in for the GRPO advantage estimator; advantages grow with search budget.
- Agon: competitive cross-model RL where two models grade each other implicitly by attempting the same problems. Doubles GRPO's pass@1 on the hard split of DeepMath with Qwen3, roughly eight times the gain of an untrained Mixture-of-Agents baseline.
- SCOPE (self-play): data-free co-evolving Challenger/Solver policies with a frozen self-judge that writes task-specific rubrics. Matches GRPO trained on ~9K curated prompts across 7–8B models on open-ended benchmarks.
- CORE (Contrastive Reflection): non-parametric; compares successful and unsuccessful traces to generate natural-language reasoning insights. Outperforms GRPO under fixed rollout budgets with as few as five training samples.
Applications beyond math reasoning
While math and code are the canonical GRPO domains, the events bundle documents its application across a wide surface:
- Mobile GUI agents: MobileGym uses GRPO on Qwen3-VL-4B-Instruct, achieving +12.8pp on a 256-task test set with 95.1% sim-to-real transfer.
- Multilingual reasoning: Luar builds selective English-translation triggering on top of GRPO, with especially large gains on low-resource languages.
- Scientific knowledge extraction: Agents-K1 trains a 4B information-extraction model with GRPO to populate a 2.46M-paper knowledge graph.
- Safety classification: GRPO with intent-faithfulness rewards yields the strongest average performance across five external safety benchmarks in the AIMS study.
- Multimodal judging: GRPO-based reward modeling improves perceptual fidelity in multimodal LLM-as-a-Judge settings.
- Deep research agents: DeepRubric-8B uses rubric-based GRPO to match prior open-source state-of-the-art on three benchmarks with ~13× fewer RL GPU-hours.
- Orchestrator training: Sakana AI's Fugu-Ultra uses GRPO (alongside SFT and evolutionary optimization) to train a model that dynamically delegates to Claude Opus 4.8, Gemini 3.1 Pro, and GPT-5.5.
Where it's heading
The evidence from this bundle points in three directions simultaneously. First, GRPO's failure modes are well-characterized enough that targeted fixes (STARE, TRIAGE, AXPO, SCOPE-RL) can be layered on without replacing the core algorithm — suggesting GRPO will remain the practical baseline for some time. Second, the compute efficiency results (QuasiMoTTo, single-layer training) suggest the cost of GRPO-style training will continue to fall. Third, the strongest challengers (Agon, VPO) suggest that group-relative advantage estimation may be the floor rather than the ceiling: as inference-time search becomes standard and competitive self-play becomes tractable, the next generation of post-training algorithms may use GRPO as a component rather than as the top-level objective.
