Beyond the GRPO name: advantage scales, clipping units, and train–rollout mismatch¶
简体中文 · English
Updated 2026-11-04 · 55 topic entries / 7 representative comparisons
The GRPO family differs in how rewards become advantages, which likelihood ratio is clipped, and which distribution produced samples. Dr. GRPO, DAPO, GSPO, and TIS/IcePop modify different parts; not every ratio is the same off-policy correction.
Mechanisms · Trade-offs · Cross-paper questions · Updates
How to read this card
The table compares representative methods. It is not a shared-benchmark leaderboard. Teaching examples are not measured results.
Mechanism comparison¶
Axes: Advantage scale: group mean, group standard deviation, and length normalization · Update unit: token surrogate or sequence ratio · Sampling validity: zero-variance groups and dynamic resampling · Distribution roles: current training policy, old training policy, and rollout engine
| Paper and source | Mechanism | Benefits and trade-offs |
|---|---|---|
| RLiD: Reinforcement Learning in the Decision Plane for Language Agents and LLMs · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 结果、负控与边界 |
Changes the optimization event from free text to executable menu-action probabilities, with single-pass label readout and decision PPO. Marginalizing realizations within one policy removes their variance. | Menus, memory, and afterstates change the interface. The identity neither compares different models nor removes temporal-credit noise. Local work is probability/PPO/enumeration auditing only; full multi-environment training is not reproduced because of cost. |
| DeepSeekMath / GRPO · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Replaces a critic with same-question group reward mean and standard deviation, retaining old-policy ratios, clipping, and reference KL. | Avoids a value model but needs multiple responses per question. Uniform rewards give no relative direction. Local complete-candidate actions are not DeepSeekMath training. |
| Dr. GRPO · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Removes within-response length averaging and group standard deviation but keeps group centering, examining implicit difficulty and length weights. | Removing normalizers changes relative gradient scales without creating signal in all-wrong groups. Local synthetic lengths are diagnostic only. |
| DAPO · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Combines Clip-Higher, zero-variance group filtering, token-level loss, and overlong reward handling. | Count effective training samples separately from actual generation budget. Local pseudo-tokens do not reproduce distributed long-CoT training. |
| GSPO · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Averages response log-ratios before exponentiating and clips the entire response with a shared sequence ratio. | Changes how anomalous tokens affect the trajectory, not the reward itself. Local work has no actual token trajectories or MoE routing. |
| TIS · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Corrects training versus rollout engine probabilities and caps only large weights from above. | Retains small ratios and limits heavy tails, but does not replace stale-policy correction. Local deterministic engine perturbations verify gradient behavior. |
| IcePop · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Uses a two-sided validity interval for engine ratios, retaining the original ratio inside and zeroing policy gradients outside. | Drops severely mismatched tokens instead of clamping all ratios to interval boundaries. No real MoE cluster was run locally. |
Trade-off analysis¶
Identify the numerator and denominator before discussing stability¶
GSPO aggregates response log-ratios of current versus old policy. TIS and IcePop compare probabilities under the same weights across training and rollout engines. One controls update size and the other corrects implementation mismatch. A training loop can need both; neither automatically covers the other's bias.
Normalization changes weights, not missing signals¶
GRPO standard deviation, its removal in Dr. GRPO, and DAPO aggregation over valid tokens alter sample contributions. An all-right or all-wrong group can still have no direction after centering. Dynamic sampling controls whether such groups reach updates. Count discarded generation work, not just optimizer steps.
Sources: DeepSeekMath / GRPO · Dr. GRPO · DAPO
Choose the update event before clipping: text and decisions are different variables¶
GRPO/GSPO mainly change advantages or ratio units of an existing text policy. RLiD first changes the probability event to a compiled menu action. Its Rao–Blackwell account conditions on alternative wordings of the same action within one text policy; it does not prove a separately trained LDM stronger. Afterstate menus, memory provenance, label calibration, and interface matching must be specified first. Sharing a PPO formula does not remove these differences.
Sources: RLiD: Reinforcement Learning in the Decision Plane for Language Agents and LLMs · DeepSeekMath / GRPO · GSPO
Cross-paper questions¶
Try answering before expanding the explanation. Each question relates at least two papers.
Why does lower variance after text-to-action marginalization not imply RLiD wins every task over reasoning-token policies?¶
Show answer and example
Answer: The result fixes one text policy and action-dependent advantages, preserving expected updates while removing realization variance. Real LDMs have different parameterizations and training paths, while reasoning adds computation. Temporal credit and menu interfaces remain separate. The paper's Qwen3.5-2B ARC result still favors token policies; that counterexample must be retained.
Illustrative example: Teaching example: Two wordings of open from one policy can be marginalized. Replacing it with another label-only network lies outside that same-policy identity.
Sources: RLiD: Reinforcement Learning in the Decision Plane for Language Agents and LLMs · DeepSeekMath / GRPO
If all four responses are wrong, does changing GRPO to Dr. GRPO create a learning direction?¶
Show answer and example
Answer: Not when rewards remain identical: removing standard deviation leaves centered advantages zero. Dr. GRPO changes scale bias; DAPO filters such groups and samples again. If budget never yields distinguishable outcomes, improve exploration or supervision rather than relabel normalization.
Illustrative example: Teaching example: With all rewards zero, subtracting the mean still yields zero. A constant denominator does not make one trajectory better.
Sources: DeepSeekMath / GRPO · Dr. GRPO · DAPO
Is GSPO equivalent to clipping each token with PPO and then averaging?¶
Show answer and example
Answer: No. GSPO aggregates log-ratios into one response ratio before clipping. Per-token clipping applies nonlinear selection first. Averaging and clipping do not commute, so outlier tokens have different effects. Sequence rewards do not mathematically require one unique surrogate.
Illustrative example: Teaching example: With one very high ratio and other ratios near one, per-token clipping can clip just that token; GSPO assigns the aggregate weight to the entire response.
Sources: DeepSeekMath / GRPO · GSPO · DAPO
How do TIS and IcePop handle a very small engine ratio, and is that policy staleness?¶
Show answer and example
Answer: TIS caps only large ratios and retains the small one. IcePop zeros its policy gradient below the lower bound. Both address engine mismatch, not drift between current and old sampling weights. Masking is not standard PPO clipping or a cure for stale policies.
Illustrative example: Teaching example: Different kernels giving different probabilities at identical weights is an engine issue. Sampling from weights several updates old introduces separate policy staleness.
Update log¶
Updates record new approaches, changed trade-offs, and follow-up evidence.
2026-11-04 · Reviewed RLiD and added the text-event versus executable-action axis. Same-policy variance marginalization is not superiority of differently trained LDMs; label-marginal KL also differs from row entropy. L1 components only; costly full training is not reproduced.
Affected: RLiD: Reinforcement Learning in the Decision Plane for Language Agents and LLMs
2026-10-27 · Initial edition: compares GRPO/Dr. GRPO/DAPO advantages and normalization, GSPO sequence ratios, and TIS/IcePop engine correction; 54 abstracts reviewed and six papers read closely.
Covers 54 topic entries; see the full list below.
All 55 topic entries
The full catalog follows. The main comparison covers the representative papers listed above.
- RLiD: Reinforcement Learning in the Decision Plane for Language Agents and LLMs
- MOER: Mix-Policy Optimization under Expert Prefix Guidance with Selective Rollback
- CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents
- On-Policy Residual Learning for Diffusion Language Models
- Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards
- Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL
- Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence
- When KL Regularization Misfires in Group Policy Optimization
- Sharpening Tax in Post-Training
- RLCPR: Rethinking Probability-Based Reinforcement Learning From Posterior Concentration
- Range-GRPO: Policy Optimization via Pairwise Relations among Reward Intervals
- Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis
- Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
- Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR
- Turn-level Multiscale Density Ratio Estimation for LLM Agents
- MaD-RL: Matching Distributions for Calibrating LLMs with Reinforcement Learning
- Beyond Confidence: Stability-Aware Test-Time Adaptation for LLM Reasoning
- Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation
- TTPO: Test-Time Policy Optimization
- Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms
- From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection
- Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization
- GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs
- Beyond On-Policy Exploration: Integrating External Policy Rollouts for Reinforcement Learning in Diffusion Language Models
- ReCo
- Reinforcement Learning for Code Optimization
- Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information
- ISO: An RLVR-Native Optimization Stack
- RIPO
- Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment
- STARE
- KPop
- GPRL
- ConSPO
- Missing Old Logits
- LUSPO
- Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learning
- Online IcePop
- Stabilizing RL with LLMs
- IcePop
- GPPO
- TIS
- GSPO
- Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty
- CISPO / MiniMax-M1
- TTRL
- LUFFY
- VAPO
- Dr. GRPO
- DAPO
- REINFORCE++
- RLOO
- DeepSeekMath / GRPO
- ReMax
- InstructGPT / PPO-RLHF