Reward provenance determines where credit belongs

简体中文 · English

Updated 2026-10-30 · 37 topic entries / 7 representative comparisons

Process labels, final outcomes, self-confidence, and hindsight attribution are distinct evidence. Stepwise PRMs, MRTPO, TACO, and MART-GRPO obtain labels, localize reward, damp positive credit, and filter failed suffixes respectively. INTUITOR uses an intrinsic distributional signal, not external verification. CONCUR instead prioritizes reward dimensions by projecting gated auxiliary advantages onto a primary-reward order cone; it does not assign token credit over time.

Mechanisms · Trade-offs · Cross-paper questions · Updates

How to read this card

The table compares representative methods. It is not a shared-benchmark leaderboard. Teaching examples are not measured results.

Mechanism comparison

Axes: Evidence: human step labels, successful continuation rate, verifiers, and intrinsic distributions · Credit granularity: steps, tokens, reward dimensions, and action chunks · Changes: add supervision, redistribute, downweight, or mask · Visibility: generation-time information versus hindsight

Paper and source	Mechanism	Benefits and trade-offs
Let's Verify Step by Step · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Trains a stepwise process reward model and prioritizes labeling uncertain steps.	Depends on annotated step coverage. Local candidate diagnostics do not reproduce the original large-scale process evaluation.
Math-Shepherd · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Samples continuations from an intermediate step and uses final correctness frequency as its automatic label.	Reduces human annotation but depends on the continuation policy and sample budget. Original scale was not reproduced.
Each Reward to Its Relevant Tokens: Reward-Specific Token-Level Credit Assignment for Multi-Reward RL · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 结果、代码映射与边界	Replays one fixed sampling policy with full verifier context and leave-one-block-out context, using full-vocabulary JSD to locate tokens relevant to each reward.	Preserves total credit but depends on information blocks. Full SQL/tool experiments were too costly and not reproduced; current results are an L1 gradient diagnostic.
TACO · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Reduces only positive advantages for high local tail-risk tokens while retaining negative advantages.	Avoids reinforcing local errors from a successful trajectory. The local surprisal proxy does not reproduce the contextual risk predictor.
When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 高成本完整实验	Uses a hindsight VLM to locate the last unrecovered error and masks later credit while retaining group advantages and the original denominator.	Hindsight boundaries need not satisfy ideal stopping-time assumptions. Full robot/VLM training was too costly; A100 evidence verifies only the L1 component.
INTUITOR · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Uses length-averaged per-token full-vocabulary KL(U∥policy) as intrinsic self-certainty reward.	No external verifier is needed, but confidence does not imply correctness. The local candidate diagnostic uses the opposite KL direction and does not implement the defining reward.
CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 高成本完整实验	Standardizes within groups, projects gated auxiliary scores onto the closest order-preserving solution, handles ceiling constraints according to T/A strategies, and whitens across the batch.	Ties below the primary ceiling need not be equal and projected scores are not re-standardized. Ungated dimensions can still reverse rankings. Full 32-GPU training was not reproduced; local work is an advantage estimator.

Trade-off analysis

Automatic process labels are not unconditional step truth

Human supervision asks whether a particular step is valid. Math-Shepherd estimates whether sampling from it eventually yields a correct answer, reflecting the continuation policy's ability to recover and finish. INTUITOR reads only model distributions, without final-answer correctness. Distinguish label meaning before comparing cost.

Sources: Let's Verify Step by Step · Math-Shepherd · INTUITOR

Credit filters act on different quantities and preserve different invariants

TACO attenuates positive advantages only. MART-GRPO masks failed-episode credit after a hindsight error boundary without changing the denominator. MRTPO redistributes signed credit across verifier information blocks with total and magnitude constraints. Deleting suspicious tokens and re-averaging cannot reproduce them all.

Sources: TACO · When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning · Each Reward to Its Relevant Tokens: Reward-Specific Token-Level Credit Assignment for Multi-Reward RL

Reward priorities and token credit use different axes

TACO lowers positive credit within one response by local risk and preserves negative credit. CONCUR compares primary and auxiliary rewards across responses to the same prompt, adjusting standardized auxiliary order and ceiling differences. It does not identify a faulty token. A scalar loses both the nearest projection and sign separation. A ceiling tie removes that auxiliary's ordering for the group, not gradient training entirely.

Sources: CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning · TACO

Cross-paper questions

Try answering before expanding the explanation. Each question relates at least two papers.

Is a low Math-Shepherd continuation success rate equivalent to a human PRM finding a logically wrong step?

Show answer and example

Answer: No. Subsequent sampling and budget affect the rate; correct steps can lead to difficult continuations. A human label evaluates the step itself. Both can help, but automatic labels are not logical ground truth. Keep training answers out of evaluated model inputs.

Illustrative example: Teaching example: A correct equation followed by arithmetic the model cannot finish can yield zero successful continuations without invalidating the equation.

Sources: Let's Verify Step by Step · Math-Shepherd

Why not implement TACO and MART-GRPO by deleting tokens and averaging again?

Show answer and example

Answer: TACO preserves negative credit and damps risky positive credit. MART-GRPO zeros a failed suffix but keeps the original valid-action-token denominator. Re-averaging amplifies remaining terms and changes the objective. MRTPO separately preserves total credit through redistribution.

Illustrative example: Teaching example: After masking the last four of ten tokens, dividing by six instead of ten makes remaining negative terms stronger than MART defines.

Sources: TACO · When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning · Each Reward to Its Relevant Tokens: Reward-Specific Token-Level Credit Assignment for Multi-Reward RL

Is a sharper INTUITOR distribution the same reward improvement as more correct Math-Shepherd continuations?

Show answer and example

Answer: No. INTUITOR uses full-vocabulary KL(U∥policy); Math-Shepherd measures final correctness. Models can be confidently wrong, and reversing KL changes the quantity. The local candidate diagnostic lacks the original direction and per-token full-vocabulary calculation, so do not rank capabilities with it.

Illustrative example: Teaching example: A model can always select one wrong number sharply while its verified continuation success remains zero.

Sources: INTUITOR · Math-Shepherd

Why can't CONCUR simply reuse TACO risk weights or reduce the length reward coefficient?

Show answer and example

Answer: TACO dampens positive token credit to avoid reinforcing local errors. CONCUR ensures a higher primary-reward response does not receive a lower gated auxiliary score and can merge ceiling scores. Scaling leaves existing inversions intact and does not give a nearest order-preserving projection.

Illustrative example: Illustration: Correctness scores 0.6 and 0.9 may have standardized length scores 1 and −1. Multiplying both length scores by 0.1 keeps the inversion, and token risk cannot order the two responses by correctness.

Sources: CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning · TACO

Update log

Updates record new approaches, changed trade-offs, and follow-up evidence.

2026-10-30 · Reviewed CONCUR auxiliary-advantage projection, ceiling ties, and batch-level trust updates separately from TACO positive token credit. Evidence covers an L1 estimator, not 32-GPU training.

Affected: CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning · TACO

2026-10-27 · Initial edition: compares human and automatic process labels, MRTPO reward localization, TACO positive credit, MART suffix masking, and INTUITOR intrinsic rewards; 36 abstracts and six close reads.

Covers 36 topic entries; see the full list below.

All 37 topic entries

The full catalog follows. The main comparison covers the representative papers listed above.

• When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning
• MGE-PO: Entropy-Aware Credit Assignment for Mathematical Reasoning
• Multi-Reward Hacking: From Aggregation-Induced Drift to Trust-Anchored Defense
• Learning to Reason with Selective Internal Feedback
• Progress Reward Model for Reinforcement Learning via Large Language Models
• OpCredit: Operation-Aware Policy Learning for Agentic Knowledge Graph Question Answering
• Each Reward to Its Relevant Tokens: Reward-Specific Token-Level Credit Assignment for Multi-Reward RL
• KnowGPO: Knowledge-Grouped Policy Optimization for Search Agents
• Training Reasoning Models to Generate Diverse Scientific Ideas
• When Pairwise Preferences Cycle: A Reliability-Anchored Preference Walk for Video GRPO
• Inject a Turn, Not a Token: Learning Where to Redirect LLM Reasoning for Self-Correction
• SOAR: Source-Aware Reinforcement Learning with Precise Credit Assignment for Search Agents
• Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards
• Token-Level Video Reinforcement Learning
• My FAULT: Self-Diagnosis as Credit Assignment in Self-Evolving Agentic Reinforcement Learning
• CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning
• T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
• SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learning
• PACT: From Credit Assignment to Critic Alignment
• TIAO: Token Importance-Aware Policy Optimization for Text Summarization
• SPEAR: Distilling Domain-Adaptive Reasoning Skeletons via Sequential Symbolic Alignment in Reinforcement Learning
• V-Rubrics: Visual Faithfulness via Rubric-Based Reinforcement Learning
• SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning
• Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization
• Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL
• RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction
• ADRS
• SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning
• CoRT
• TCR
• Co-Evolving LLM Evaluators and Policies via DynamicRubric
• TACO
• SCAR: Shapley Credit Assignment for More Efficient RLHF
• INTUITOR
• Math-Shepherd
• Let's Verify Step by Step
• CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning

Paper-by-paper questions · All deep dives