What GRPO is
GRPO — Group Relative Policy Optimization — is a training technique for making AI language models better at tasks that have checkable answers, like math problems or coding challenges. The core idea is simple: instead of asking a human "was this answer good?", you have the model attempt the same problem many times, then automatically reward the attempts that got it right and penalize the ones that didn't. The model learns by comparing its own outputs against each other — hence "group relative."
Think of it like a student who takes ten practice runs at the same exam question. The runs that led to the right answer get reinforced; the ones that went off track get discouraged. No teacher needs to grade every step — just the final result.
Why it matters
Before techniques like GRPO, improving a model's reasoning usually meant either collecting expensive human feedback or writing out detailed step-by-step demonstrations for the model to copy. GRPO bypasses both. If you can write a simple checker — "is this math answer correct?" or "does this code pass the tests?" — you have everything you need to train.
This has made GRPO enormously popular. Hugging Face's TRL library, the most widely used open-source post-training toolkit, reached version 1.0 with GRPO as a first-class supported method — a sign the technique has crossed from research novelty into standard practice. A preprint called VibeThinker reported that a 3-billion parameter model trained with supervised fine-tuning plus GRPO could match or beat Claude Opus 4.5 on reasoning benchmarks — a striking result that, if it holds up, would show just how much headroom GRPO unlocks even in small models.
How it works (the plain version)
1. Sample a group of answers. For a given question, the model generates several candidate responses — say, eight different attempts. 2. Score them. A verifier checks which answers are correct. No human needed; the checker is usually a simple rule or test suite. 3. Compute relative advantage. Answers that did better than the group average get a positive score; worse-than-average answers get a negative score. 4. Update the model. The model's weights are nudged to make the better answers more likely next time, and the worse ones less likely.
Repeat this millions of times across many problems, and the model gradually gets better at the underlying skill.
What can go wrong
GRPO is powerful but not magic. Researchers have documented several failure modes:
- Entropy collapse. The model can stop exploring — it finds one style of answer that works and repeats it, losing the diversity needed to solve harder problems. A method called STARE addresses this by selectively reweighting which tokens get credit, outperforming alternatives by 4–8% on hard math benchmarks.
- Sparse rewards in tool use. When a model uses external tools (like a database or code interpreter) and the tool returns cryptic feedback (like empty brackets
[]), GRPO can't learn from it. One paper documented a "peak-then-collapse" pattern where tool-use rates rose then fell to zero within 50 training steps. - The Thinking-Acting Gap. In agentic settings — where the model needs to both reason and take actions — standard GRPO training leads to tool use in only about 30% of attempts. A method called AXPO fixes this by resampling tool calls for failed attempts, gaining 1.8 percentage points over standard GRPO.
The growing ecosystem of improvements
Because GRPO is now so widely used, it has become the baseline that new research improves upon. A few notable directions from the events bundle:
- Better rewards. SCOPE-RL adds intermediate reward signals before and after each step, improving accuracy by up to 11 percentage points and cutting reasoning tokens by 27% over standard GRPO. DeepRubric builds structured evaluation rubrics from evidence trees, achieving comparable results to prior state-of-the-art with 13× fewer GPU-hours.
- Better exploration. POPE (from Carnegie Mellon) gives the model partial solution hints during training so it can discover complete solutions it would otherwise never find — improving HMMT 2025 pass@1 from 31.0% to 37.8%.
- Cheaper training. QuasiMoTTo replaces independent random samples with correlated quasi-Monte Carlo samples, achieving equivalent GRPO training performance with 50% fewer steps. A separate finding showed that training only a single transformer layer — specifically, layers in the middle of the stack — can recover most of the gains from full-parameter GRPO training.
- Diversity over accuracy. Vector Policy Optimization (VPO) replaces GRPO's single reward score with a vector of rewards, explicitly training the model to produce diverse solution sets — useful when you plan to run many attempts at test time and pick the best.
Where GRPO is used today
The technique has spread well beyond math. The events in this bundle show GRPO being applied to:
- Robotics and mobile agents (MobileGym: +12.8 percentage points on a 256-task mobile GUI benchmark)
- Multilingual reasoning (Luar: selectively invoking translation only when needed)
- Multimodal vision (AXPO, ATLAS, VeriEvol)
- Agentic tool use (PROVE, TRIAGE, ReuseRL)
- Safety classification (AIMS dataset: GRPO with intent-aware rewards outperforms SFT and DPO)
- Scientific knowledge extraction (Agents-K1: a 4B information-extraction model trained with GRPO)
- Orchestrating multiple AI models (Sakana AI's Fugu-Ultra uses GRPO to learn which underlying model to delegate each subtask to)
Where it's heading
The research frontier is moving in two directions at once. One direction pushes GRPO further — fixing its failure modes, making it cheaper, applying it to harder domains. The other direction asks whether GRPO is the right foundation at all: competitive self-play frameworks like Agon (which doubled GRPO's pass@1 on hard math by having two models grade each other) and data-free self-play frameworks like SCOPE (which matched GRPO without any curated training prompts) suggest the next generation of reasoning training may look quite different. For now, though, GRPO remains the workhorse — the method every new idea is measured against.
