Reward and credit: which steps deserve a good final score?¶
简体中文 · English
LLM post-training / 148 questions · 37 papers
Try answering each question before opening the explanation. The bank grows as papers and questions are added.
Read this topic's comparisons, analysis, and cross-paper Q&A
148 questions · 37 papers
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why mask late feedback on MART-GRPO failures?
When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning
Show answer and example
Answer: Terminal failure doesn't mean every later action caused it. Find the last unrecovered error and mask subsequent negative credit while retaining relevant earlier signal; execution itself is not stopped early.
Illustrative example: A failed regrasp after a drop needn't be punished equally solely because the episode failed.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why keep original advantages and denominator after masking penalties beyond the last unrecovered error?
When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning
Show answer and example
Answer: Masking isolates where failure credit applies. Changing group rewards/advantages or denominator adds other effects. Execute whole rollout before VLM hindsight; that does not guarantee stopping-time theory applies.
Illustrative example: Mask idle behavior after a mishap without amplifying punishment for the remaining actions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
Why isn't a retrospective VLM boundary automatically a theoretical stopping time?
When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning
Show answer and example
Answer: The VLM sees future trajectory, unlike a decision based only on the past. Arbitrary masks don't inherit unbiasedness or variance reduction guarantees.
Illustrative example: Choosing an important step after seeing the outcome differs from deciding as it happens.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
May MART recompute advantages or renormalize by unmasked tokens?
When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning
Show answer and example
Answer: No. Keep original group advantages and valid-action denominator; apply the mask only to feedback. Valid interval includes its upper endpoint. Invalid format/identity/bounds are errors, not semantic-uncertainty fallback.
Illustrative example: Shrinking the denominator after masking amplifies retained gradients and changes the method.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Are MGE-PO success/failure and high/low entropy the same label?
MGE-PO: Entropy-Aware Credit Assignment for Mathematical Reasoning
Show answer and example
Answer: No. Verified outcome and policy uncertainty are distinct axes; all four combinations occur. Entropy alone is not correctness.
Illustrative example: A model can confidently be wrong or hesitate and get it right.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why use generation-time cached entropy rather than recalculate after an update?
MGE-PO: Entropy-Aware Credit Assignment for Mathematical Reasoning
Show answer and example
Answer: Credit describes uncertainty at sampling. New policy entropy would mix policy changes into old trajectory judgment. Combine outcome and entropy quadrants with separate curriculum/KL; truncated replies can still shift through recentering.
Illustrative example: A reply becoming more certain after training was not necessarily sampled confidently.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How do MGE-PO quadrants differ from rewarding low entropy alone?
MGE-PO: Entropy-Aware Credit Assignment for Mathematical Reasoning
Show answer and example
Answer: Combine outcome with entropy; confidence is not inherently good. Low-entropy reward alone can reinforce confident errors. Verifiable correctness is needed without leaking evaluation gold.
Illustrative example: Low-entropy correct and low-entropy wrong replies need different credit.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Why can a truncated response have nonzero final advantage after zero direct correction?
MGE-PO: Entropy-Aware Credit Assignment for Mathematical Reasoning
Show answer and example
Answer: Zero direct increment doesn't prevent batch recentering from moving all advantages. Use lagged problem EMA for curriculum and cached sampling-policy entropy, not future generations.
Illustrative example: Positive corrections on peers shift the mean and move an uncorrected reply relatively.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why choose a reliable anchor before TAG multi-reward training?
Multi-Reward Hacking: From Aggregation-Induced Drift to Trust-Anchored Defense
Show answer and example
Answer: Auxiliary objectives can be gamed while harming the main goal. Enable auxiliary credit only when the anchor is met; this differs from shrinking its coefficient.
Illustrative example: A short incorrect answer shouldn't get positive brevity credit on its own.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why discard anchor probes and restart, and when do auxiliary rewards apply?
Multi-Reward Hacking: From Aggregation-Induced Drift to Trust-Anchored Defense
Show answer and example
Answer: Separate probes measure reward-induced drift, not supply final weights. Restart from the common baseline to isolate selection. Anchor failure blocks auxiliary credit. Sample under each probe to preserve KL direction.
Illustrative example: Brevity can't raise credit on an incorrect answer if correctness is the anchor.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does TAG's low-drift probe selection differ from manually selecting correctness?
Multi-Reward Hacking: From Aggregation-Induced Drift to Trust-Anchored Defense
Show answer and example
Answer: Probe each reward from the same baseline, choose via probe-to-baseline reverse KL, discard probes and retrain. A manually chosen anchor skips that process.
Illustrative example: Fast reward growth can cause large drift, not make it the chosen anchor.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Why estimate TAG drift on each probe's own trajectories?
Multi-Reward Hacking: From Aggregation-Induced Drift to Trust-Anchored Defense
Show answer and example
Answer: Reverse-KL expectation is under the probe. Baseline sampling changes direction. Pool all generated tokens, not per-response means. A finite-sample estimate may be negative; do not clamp.
Illustrative example: Long and short responses contribute their tokens jointly rather than equal response weights.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why not treat mean self-certainty as a correct label in SIF?
Learning to Reason with Selective Internal Feedback
Show answer and example
Answer: Confidence may grow for wrong answers. Compare ranked certainty windows across same-prompt replies, then use two-level voting for relative reward. This is not independent truth.
Illustrative example: A confident error does not become right due to a high mean.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why compare sorted certainty windows instead of using entropy or high-probability tokens as labels?
Learning to Reason with Selective Internal Feedback
Show answer and example
Answer: Mean confidence can amplify errors. Build a profile using uniform-to-policy KL, then compare eligible same-prompt replies at window and peer levels. This is not entropy or sampled probability; filter ineligible/equal-reward groups and share policy/KL masks.
Illustrative example: Multiple windows distinguish a few extreme positions from broadly uniform certainty.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
Why isn't SIF's KL(U||π) negative entropy?
Learning to Reason with Selective Internal Feedback
Show answer and example
Answer: It averages log probabilities uniformly; negative entropy weights by policy probabilities. Direction and low-probability sensitivity differ, as does using only sampled tokens.
Illustrative example: A rare token approaching zero affects uniform-to-policy KL differently from policy-weighted entropy.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Should ineligible SIF replies disable KL as well?
Learning to Reason with Selective Internal Feedback
Show answer and example
Answer: Yes. Policy and KL use the same selection mask. Keep original response normalization, eligibility, strict comparisons and sample SD. A tie isn't a win.
Illustrative example: Turning off policy loss but retaining KL would still update a filtered response.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why does PRM4RL provide subtask semantics and a progress potential?
Progress Reward Model for Reinforcement Learning via Large Language Models
Show answer and example
Answer: Subtask meaning enriches state with current phase; potential differences supply dense progress despite sparse terminal success. One changes inputs, the other rewards, and planning can be offline.
Illustrative example: A robot knows it is grasping and receives progress for moving closer.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why use discounted potential differences and not treat timeout as termination?
Progress Reward Model for Reinforcement Learning via Large Language Models
Show answer and example
Answer: Neighbor terms telescope to endpoints. Correct absorbing boundary prevents terminal potential changing the objective. A timeout may leave future progress possible, so don't zero it automatically. Subtask encoding and shaping remain independent.
Illustrative example: Zeroing potential when time expires near a goal can fabricate a progress penalty.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does potential shaping differ from rewarding raw progress at each step?
Progress Reward Model for Reinforcement Learning via Large Language Models
Show answer and example
Answer: Discounted differences compare transitions and telescope, while repeated high potential can reward idling. Terminal conditions matter; differencing alone does not guarantee policy invariance.
Illustrative example: Staying at a high-progress state without finishing can accrue reward under raw per-step scores.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Why distinguish true termination from time truncation in PRM4RL?
Progress Reward Model for Reinforcement Learning via Large Language Models
Show answer and example
Answer: An absorbing terminal can have zero potential; time-limited states can retain successor value. Use the formula absent explicit termination and review generated functions.
Illustrative example: Timeout with a resumable task differs from actual success or failure.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why assign OpCredit search credit to individual operations?
OpCredit: Operation-Aware Policy Learning for Agentic Knowledge Graph Question Answering
Show answer and example
Answer: Successful trajectories can detour and failed ones can explore usefully. Branch outcomes and frontier changes score local operations, then align generated reasoning/action tokens, not environment observations.
Illustrative example: Finding a key entity can be useful even if the final answer fails.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why match visited set, remaining budget and phase when comparing operations?
OpCredit: Operation-Aware Policy Learning for Agentic Knowledge Graph Question Answering
Show answer and example
Answer: The same entity can have different future opportunities. BOC matches context and deduplicates by trajectory; FTC uses training answer neighborhoods for local improvement. Keep answers out of actor input and observations out of policy loss.
Illustrative example: Two visits to one node differ if only one has explored its neighbors.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
What does BOC branch comparison use versus FTC frontier progress?
OpCredit: Operation-Aware Policy Learning for Agentic Knowledge Graph Question Answering
Show answer and example
Answer: BOC reuses outcomes from matched states; FTC assesses local frontier against a training-only answer neighborhood. Training labels can score credit afterward but remain hidden during sampling. They are distinct signals.
Illustrative example: Closeness to the training answer may score a neighbor after rollout, never enter the search request.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Why not match OpCredit branches by active entity alone?
OpCredit: Operation-Aware Policy Learning for Agentic Knowledge Graph Question Answering
Show answer and example
Answer: Match problem, visited set, budget and phase to compare similar opportunities. Count each trajectory once. FTC uses the frontier's best chance, not graph-wide mean distance. Detach credit and reference.
Illustrative example: Arriving with no budget differs from arriving with resources to search.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why not broadcast every MRTPO reward equally over a response?
Show answer and example
Answer: Format, executability and correctness can depend on different tokens. Removing one information block and measuring distribution shift identifies likely relevant positions.
Illustrative example: SQL grammar and numerical correctness may point to different tokens.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why compare vocabulary-wide JS after removing each reward block?
Show answer and example
Answer: A sampled token may miss movement among other candidates. Full-distribution changes associate reward information with positions. Mean-preserving weights apply to both signs with bounded magnitude. Verifier context belongs only in training replay.
Illustrative example: Removing SQL-format feedback may shift mass between comma and parenthesis while the sampled token is unchanged.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does information ablation differ from sampled-token probability changes?
Show answer and example
Answer: Use full-vocabulary JS under the same frozen policy. A single token can miss mass redistribution. Association is not causal correctness, and reward info cannot enter original student sampling.
Illustrative example: The sampled token may stay constant while alternative probabilities move.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Why use uniform ones when all MRTPO relevance is zero?
Show answer and example
Answer: If localization is unavailable, uniform weights retain total credit. Keep response mean one before second-moment limits and batch normalization. Zeros delete the objective.
Illustrative example: An empty format block should not erase the format advantage.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why group KnowGPO steps by known fact sets?
KnowGPO: Knowledge-Grouped Policy Optimization for Search Agents
Show answer and example
Answer: Different queries/pages can yield the same knowledge. Verified atomic conditions build a monotone state, enabling comparisons of what actions add from a common start.
Illustrative example: Different wording can establish the same person and year and share a pre-state.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why group by pre-action knowledge rather than post-action knowledge?
KnowGPO: Knowledge-Grouped Policy Optimization for Search Agents
Show answer and example
Answer: Use shared initial facts to compare action benefits without conditioning on gains. A training verifier monotonically accumulates facts; combine process/terminal advantages for generated tokens only, hiding checklist from the actor.
Illustrative example: Grouping a successful and unsuccessful search by resulting facts separates the two outcomes before comparison.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
Why not group KnowGPO by state after the action?
KnowGPO: Knowledge-Grouped Policy Optimization for Search Agents
Show answer and example
Answer: Credit compares actions from a common starting point. Grouping by outcome separates actions that add different facts. Follow algorithmic pre-state despite prose ambiguity.
Illustrative example: An action finding a new fact should be compared to a miss from the same known facts.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
How handle KnowGPO answer steps and tool observations in loss?
KnowGPO: Knowledge-Grouped Policy Optimization for Search Agents
Show answer and example
Answer: Answer steps skip process grouping but keep the terminal mixture term. Advantage covers generated reasoning/actions only. Keep verifier facts and tool observations out of actor targets. Singleton process advantage is zero.
Illustrative example: Environment search results are not model-generated tokens.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why generate a whole idea set instead of sampling independently and deduplicating?
Training Reasoning Models to Generate Diverse Scientific Ideas
Show answer and example
Answer: Later ideas see predecessors and can seek genuinely different directions. Reward combines quality and dissimilarity. Different strings do not prove scientific novelty; semantic judging matters.
Illustrative example: A second approach can cover a gap rather than paraphrase the first.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why preserve generated order when assigning DIVERGE diversity credit?
Training Reasoning Models to Generate Diverse Scientific Ideas
Show answer and example
Answer: Each idea sees preceding ideas, so nearest predecessor reflects its actual choice. Reordering changes credit. Evaluation's discounted reordered metric is separate; semantic distance and quality need independent judging.
Illustrative example: A second paraphrase deserves lower novelty under the original order; rearranging does not rewrite its training reward.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
Can DIVERGE training set reward and evaluated Creative Utility be exchanged?
Training Reasoning Models to Generate Diverse Scientific Ideas
Show answer and example
Answer: Training uses distance to the closest predecessor in generation order; evaluation can reorder, discount and normalize. Greedy evaluation is only a lower bound on maximum CU.
Illustrative example: Moving the best idea first can alter evaluation but doesn't change original training order.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Which fields count toward DIVERGE length penalties?
Training Reasoning Models to Generate Diverse Scientific Ideas
Show answer and example
Answer: Count tokens in problem_gap, core_claim and method, not inspiration or think, using the intended tokenizer. Schema failure gives zero without a judge call. Character count isn't a substitute.
Illustrative example: A long inspiration citation isn't part of the specified length term.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why retain opponent identity and preference cycles in RAP-Walk?
When Pairwise Preferences Cycle: A Reliability-Anchored Preference Walk for Video GRPO
Show answer and example
Answer: Win rate loses who was beaten and where preferences conflict. Edge clarity and cycle reliability inform graph credit. Cycles can reflect multi-dimensional tradeoffs, not poor samples.
Illustrative example: A can win on motion, B visuals and C adherence, forming a meaningful cycle.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why keep cyclic preferences instead of removing or labeling their videos bad?
When Pairwise Preferences Cycle: A Reliability-Anchored Preference Walk for Video GRPO
Show answer and example
Answer: Cycles can come from different strengths. Preserve opponent identities, edge clarity and cycle inconsistency, propagate reliably from losers to winners and adjust win-rate credit. Keep direction, convergence and entropy protocol; ordinary PageRank differs.
Illustrative example: A beats B, B beats C and C beats A across visual quality, motion and adherence; deleting all loses comparisons.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does reliable random walk differ from averaging wins?
When Pairwise Preferences Cycle: A Reliability-Anchored Preference Walk for Video GRPO
Show answer and example
Answer: Propagate from losers to winners with reliability, obtain stationary credit and mix with a win-rate anchor. Minimum entropy prevents excessive concentration. Average wins lack that propagation.
Illustrative example: Beating opponents of different reliability does not yield identical propagated credit.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
May nonconvergent RAP-Walk automatically add PageRank teleportation?
When Pairwise Preferences Cycle: A Reliability-Anchored Preference Walk for Video GRPO
Show answer and example
Answer: No. Report failure per protocol. Use W_ji orientation, restandardize after mixing and detach preference graph/advantages. Teleportation changes stationary credit.
Illustrative example: Reversing loser-to-winner edges can reward opposite nodes even with a normalized matrix.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Does UpTurn learn token edits or where to restart correction?
Inject a Turn, Not a Token: Learning Where to Redirect LLM Reasoning for Self-Correction
Show answer and example
Answer: It learns paragraph cut and repair gate with frozen reasoner/encoder. Keep the prefix, insert a fixed new user turn and continue with the reasoner.
Illustrative example: Restart after a bad derivation section instead of changing old token probabilities.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why learn paragraph cuts rather than fine-tune UpTurn's whole reasoner?
Inject a Turn, Not a Token: Learning Where to Redirect LLM Reasoning for Self-Correction
Show answer and example
Answer: Freezing isolates when to insert a correction; rewards come from actual regenerated suffixes at different cuts. Keep prefix, close turn, insert fixed user turn, and obey leftover budget. It isn't whole rewrite or model training. Gate cannot use gold.
Illustrative example: Retry from a paragraph boundary with less remaining budget, not unlimited free retries.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does a new dialogue turn differ from appending reflection to the assistant message?
Inject a Turn, Not a Token: Learning Where to Redirect LLM Reasoning for Self-Correction
Show answer and example
Answer: Role and conditioning boundaries change. End the assistant turn, add a new user instruction and continue within remaining budget. String append doesn't imply equivalent chat semantics.
Illustrative example: A user's request to recheck differs from the assistant continuing to reflect.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Why not give UpTurn the full L_max again after cutting?
Inject a Turn, Not a Token: Learning Where to Redirect LLM Reasoning for Self-Correction
Show answer and example
Answer: Prefix already consumes budget; suffix gets L_max minus cut position. No parseable answer means gate zero, threshold fixed. Some mechanism seeds still fail; execution is not stable success.
Illustrative example: Full budget after a late cut gives treatment extra generation cost.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why require explicit source claims in SOAR answers?
SOAR: Source-Aware Reinforcement Learning with Precise Credit Assignment for Search Agents
Show answer and example
Answer: Correctness alone doesn't verify evidential dependence. Record internal recall and external retrieval separately, then test whether the claimed stream supports the answer.
Illustrative example: A retrieval claim must be supported by retrieval records, not by a lucky memory.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why ignore an answer found in the nondeclared evidence stream?
SOAR: Source-Aware Reinforcement Learning with Precise Credit Assignment for Search Agents
Show answer and example
Answer: A declared source should actually support the answer; cross-stream rescue would reward invented dependency. Separate recall/retrieval, keep old reasoning and external evidence out of recall, and exclude observations from actor actions.
Illustrative example: If a reply cites retrieval but only recall contains the fact, provenance is inconsistent.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
Why can't SOAR internal recall see prior reasoning or retrieval?
SOAR: Source-Aware Reinforcement Learning with Precise Credit Assignment for Search Agents
Show answer and example
Answer: The audit distinguishes existing knowledge from external findings. Isolate recall before external evidence; contamination invalidates source matching.
Illustrative example: Recalling a fact after reading a webpage isn't independent memory.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Can SOAR award matching credit from the other evidence stream?
SOAR: Source-Aware Reinforcement Learning with Precise Credit Assignment for Search Agents
Show answer and example
Answer: No. Check only the declared stream. Tool observations and recall text aren't actor-loss actions. Explicitly normalize reward before group advantage; mechanism execution isn't external capability.
Illustrative example: If an answer claims recall but only a webpage supports it, correctness doesn't earn source credit.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why turn teacher–student probability residuals into local advantages?
Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards
Show answer and example
Answer: Teacher differences supply dense guidance. Subtract the student-distribution expectation to get relative local credit, then combine with verifier advantages rather than only terminal teacher scores.
Illustrative example: A teacher preferring another word at the same prefix suggests a local update.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why subtract expected residual under the student and then center per response?
Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards
Show answer and example
Answer: Raw differences depend on student coverage. Student expectation yields relative local credit; response centering separates positional effects. Add to verifier advantage, not replace it. Fix scoring across the batch and refresh teacher only next round.
Illustrative example: One word's positive residual isn't sufficient if alternatives have similarly positive residuals.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How do residual advantages differ from full-distribution KL distillation?
Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards
Show answer and example
Answer: Bound and center local advantages to feed policy updates, while KL directly constrains distribution differences. Gradients and reward combination differ despite using teacher probabilities.
Illustrative example: Subtract student expectation and response mean rather than copying logits.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Why can't Co-RA change teachers midway through a scored batch?
Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards
Show answer and example
Answer: Student/teacher consume the same scored batch. Teacher LoRA uses fixed verifier advantages, and the new teacher is used next round. Relabeling midway changes targets and provenance.
Illustrative example: Old teacher on the first half and updated teacher on the second is not fixed supervision.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why use gradient sensitivity for video-token credit in TVRL?
Token-Level Video Reinforcement Learning
Show answer and example
Answer: Terminal video reward is coarse. Sensitivity through a frozen evaluator localizes group advantage. Sensitivity means response to perturbations, not verified correctness.
Illustrative example: Action terms may affect video reward more than connective words.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why interpret high VLM-reward gradient as sensitivity rather than correct credit?
Token-Level Video Reinforcement Learning
Show answer and example
Answer: It measures local response of the current reward function. Mean-normalized token reweighting controls scale, but shortcut-driven sensitivities need not be causal. Local checks neither train videos nor prove perception gains.
Illustrative example: A corner watermark can receive a large gradient without being a route to better video quality.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does TVRL sensitivity differ from uniformly broadcasting advantage?
Token-Level Video Reinforcement Learning
Show answer and example
Answer: Normalized gradient norms emphasize sensitive positions while uniform credits ignore them. Quality depends on the frozen VLM; gradients alone are not causal evidence.
Illustrative example: Evaluator sensitivity to a word can be a shortcut.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Does local TVRL train a video diffusion model?
Token-Level Video Reinforcement Learning
Show answer and example
Answer: No. It computes gradient-based credits and mean normalization. Real video, frozen VLM gradients and policy training remain; handle all-zero norms explicitly. Random inputs are not video capability.
Illustrative example: Hand-supplied gradient sizes validate weighting without generating videos.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why verify My FAULT self-diagnoses before pricing errors?
My FAULT: Self-Diagnosis as Credit Assignment in Self-Evolving Agentic Reinforcement Learning
Show answer and example
Answer: Self-explanations can invent causes. Only evidence-supported categories/locations enter online pricing; relative costs allocate terminal credit.
Illustrative example: A claim that the tool failed needs trace evidence before penalizing tool steps.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why verify diagnostic claims before error pricing?
My FAULT: Self-Diagnosis as Credit Assignment in Self-Evolving Agentic Reinforcement Learning
Show answer and example
Answer: Hindsight explanations can rationalize. Evidence-verified location and class drive relative online prices and conserved terminal credit redistribution. No invented success reward or jointly trained local diagnosis model.
Illustrative example: If search was correct but summary dropped a fact, don't penalize search merely because the model blames it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does conserved redistribution differ from an extra error penalty?
My FAULT: Self-Diagnosis as Credit Assignment in Self-Evolving Agentic Reinforcement Learning
Show answer and example
Answer: Conservation preserves total terminal return and changes only allocation. Extra penalties change the objective and incentives.
Illustrative example: Lower one step's credit and offset elsewhere rather than deduct from the entire return twice.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
What conservation invariant should My FAULT check?
My FAULT: Self-Diagnosis as Credit Assignment in Self-Evolving Agentic Reinforcement Learning
Show answer and example
Answer: Valid-step credits sum to original return. Mean cost and step count use the same mask. Local screening/cost/conservation does not jointly train a policy and diagnosis model.
Illustrative example: Including padding in mean cost breaks conservation on valid steps.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why take absolute token log ratios in CARM?
CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning
Show answer and example
Answer: Signed shifts can cancel across positions and conceal severe drift. Average absolute magnitude and use a response-level gate to detect cancellation risk.
Illustrative example: A sharp rise at one token and equal fall at another can average to zero.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why take absolute log ratios before response averaging?
CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning
Show answer and example
Answer: Opposing probability shifts can cancel in a normal sequence mean. Absolute averaging detects magnitude and masks large mismatch. It measures policy shift, not answer quality, with a detached gate.
Illustrative example: Opposite shifts across sentence halves produce near-zero total signed drift but large mean absolute drift.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does CARM differ from filtering on geometric ratio near one?
CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning
Show answer and example
Answer: Geometric ratios allow signed cancellation; CARM measures unsigned change and can reject internally shifted responses near ratio one. This is not quality scoring.
Illustrative example: A balanced ledger can still hide large cash movements.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Should CARM differentiate through the response gate?
CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning
Show answer and example
Answer: No. Freeze selection, measure valid response tokens only and apply the specified threshold. No Qwen3.5 training or measured stability improvement.
Illustrative example: Don't add gradients through a threshold recomputed from updated probabilities.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why use remaining steps in successful traces as T2SPO distance?
T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
Show answer and example
Answer: Successful history offers state/remaining-step support examples for a frozen in-context predictor, giving denser progress than terminal success. Failure cannot be relabeled as close to success.
Illustrative example: A state three steps from completion provides a support label.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why set successor distance to zero at success but directly zero progress at failure?
T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
Show answer and example
Answer: Success reaches the goal; failure only stops execution, so treating it as distance zero creates false progress. Score before updating successful history to avoid leakage. Handle time truncation according to GRPO/PPO.
Illustrative example: A tool-exhaustion exit is not zero steps to success and must not reward the final failure.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does distance-improvement credit complement terminal success?
T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
Show answer and example
Answer: Distance differences guide intermediate steps; terminal reward still measures outcome. Mask/scale imperfect forecasts. TabPFN prediction errors don't establish task success.
Illustrative example: An action can move toward the goal without completing it; still verify terminal result.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Does T2SPO optimizer=ppo provide a full PPO trainer?
T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
Show answer and example
Answer: No. It changes truncation semantics only, no critic/GAE. GRPO zeroes truncated credit; PPO uses predicted real successor. Fit standardization/SVD only on historical supports.
Illustrative example: Changing an option string does not create a value network.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why give SHARPO a successful same-group trajectory as teacher context?
SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learning
Show answer and example
Answer: An in-task success guides rescoring student logp within interaction segments and scales original GRPO advantage. Teacher context is training credit, not a deployment gold plan.
Illustrative example: Use a successful path to strengthen related behavior without pasting it into the inference prompt.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why use bounded scaling after rescoring segments against a successful path?
SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learning
Show answer and example
Answer: Same-group success supplies extra context at the actual prefix. Segment logp gaps express local support. Bounded factors limit amplification and preserve original GRPO direction. Success context remains training-only.
Illustrative example: Retrieval in a failed trace may resemble a successful path while synthesis diverges, so weight the two segments differently.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does bounded SHARPO multiplication differ from adding teacher advantage?
SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learning
Show answer and example
Answer: Twice sigmoid lies between zero and two, so scaling preserves sign and cannot turn zero nonzero. Additive terms can flip credit. The teacher modulates intensity rather than rewriting outcomes.
Illustrative example: A failed segment can be penalized less but not rewarded merely by teacher preference.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
What constraints apply to SHARPO segments and gold plans?
SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learning
Show answer and example
Answer: Average teacher–student gaps within matching interaction segments and broadcast there, excluding tool observations. Local segments use no gold plan or 7B training; fixtures aren't capability evidence.
Illustrative example: Mixing a tool call with a different turn's answer gives the wrong segment mean.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why tie PACT token credit to prefix value?
PACT: From Credit Assignment to Critic Alignment
Show answer and example
Answer: One terminal reward is coarse. Prefix-conditioned expected return contextualizes it; R−V approximates advantage. Exact credit needs theoretical assumptions; a learned critic is an approximation.
Illustrative example: The same failure can yield different advantages at an easy versus difficult prefix.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why recompute PACT ratios after actor update, and may targets above one be clipped?
PACT: From Credit Assignment to Critic Alignment
Show answer and example
Answer: New/old token ratios correct old returns toward the updated actor's critic target. The product can exceed one; clipping changes the objective. Use stable softplus and the prescribed update order.
Illustrative example: A more likely successful action after update should change critic weighting instead of fitting only old sampling.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
Why update actor before critic in PACT instead of fitting the old policy?
PACT: From Credit Assignment to Critic Alignment
Show answer and example
Answer: The critic should follow the updated policy. Correct observed returns with a current-token new/old ratio, not an exact full-future continuation product.
Illustrative example: Once actor actions change, replaying uncorrected old returns leaves the critic behind.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Why not send ρR above one to ordinary probability BCE?
PACT: From Credit Assignment to Critic Alignment
Show answer and example
Answer: Use the specified softplus(z)−Yz form and filter out-of-range ratios. Standard BCE may reject or alter targets outside [0,1]. Sparse nonzero local outcomes aren't reliable gains.
Illustrative example: Reward one times a ratio slightly above one naturally exceeds one; do not clamp it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why mask the source document in TIAO?
TIAO: Token Importance-Aware Policy Optimization for Text Summarization
Show answer and example
Answer: Summaries should use evidence. Comparing probabilities with and without source locates evidence-sensitive tokens for trajectory/token credit. Dependence is association, not verified truth.
Illustrative example: Removing a relevant sentence can affect a name more than a conjunction.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现对应. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why use masked-source token probability changes rather than length-based credit?
TIAO: Token Importance-Aware Policy Optimization for Text Summarization
Show answer and example
Answer: Tokens vary in evidence dependence; paired scoring supplies local clues and shapes both trajectory and token credit. Length doesn't identify provenance. Masking can itself shift distribution, so association is not factual attribution.
Illustrative example: A date and a connective at similar positions need not receive the same evidence credit.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现对应. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does TIAO dual-scale credit differ from one summary score?
TIAO: Token Importance-Aware Policy Optimization for Text Summarization
Show answer and example
Answer: Whole-summary score captures overall quality; token dependence redistributes local updates. A good aggregate doesn't supervise every token correctly.
Illustrative example: One hallucinated date in a good summary calls for localized correction.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现对应. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Which response should both TIAO views score?
TIAO: Token Importance-Aware Policy Optimization for Text Summarization
Show answer and example
Answer: Use identical response and prefixes under complete/masked source. Otherwise generation changes confound the intervention. Local dependence kernel does not train the paper's 7B model.
Illustrative example: Scoring A with source and generating B without it doesn't isolate source removal.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现对应. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why convert teacher reasoning to symbolic milestones in SPEAR?
Show answer and example
Answer: Terminal reward is sparse and neural process judges costly. Domain milestones permit ordered alignment and dense progress, if extraction preserves important reasoning.
Illustrative example: Match key equations rather than copying all teacher wording.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why align milestones with LCS-F1 instead of checking the same symbols?
Show answer and example
Answer: Identical sets can break reasoning order. LCS preserves relative order and precision/recall detect redundancy and omissions, providing dense feedback. Extraction can err; local candidate axes are not a natural-language extractor.
Illustrative example: Substituting conditions before solving differs from writing the result then inventing conditions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does LCS-F1 differ from unordered overlap?
Show answer and example
Answer: LCS respects order; F1 measures coverage and redundancy. Reordered dependencies can retain the same set yet deserve different credit.
Illustrative example: Putting a conclusion before its premises can look identical to set overlap.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Does local SPEAR extract natural-language milestones?
Show answer and example
Answer: No. Candidate process/format axes proxy milestones for LCS-F1. Teacher-text extraction is separate; known symbolic labels don't prove extraction.
Illustrative example: Supplying symbols directly cannot test extracting them from a long proof.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
What do VF, RC and IF measure in V-Rubrics?
V-Rubrics: Visual Faithfulness via Rubric-Based Reinforcement Learning
Show answer and example
Answer: Visual faithfulness, reasoning consistency and instruction following. They are separate axes, not candidates or ordered steps; identify error type before assigning reward and prefix credit.
Illustrative example: Wrong requested bullet count is IF; inventing an object is VF.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why separate vision, reasoning and instruction following instead of one score?
V-Rubrics: Visual Faithfulness via Rubric-Based Reinforcement Learning
Show answer and example
Answer: A visually correct reply can reason poorly; a coherent one may invent details. Atomic rubrics localize evidence to prefixes instead of obscuring it with terminal reward. Weights don't validate labels; local proxies aren't visual models.
Illustrative example: Correct formatting doesn't excuse a nonexistent object used as a premise.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
Can V-Rubrics dimension weights replace criterion coverage?
V-Rubrics: Visual Faithfulness via Rubric-Based Reinforcement Learning
Show answer and example
Answer: No. Weights combine detected dimensions; coverage determines which errors are seen. Raising VF cannot penalize errors none of its rubrics detect. Objective tradeoff and measurement reliability differ.
Illustrative example: Object-type-only checks may miss two becoming three regardless of VF weight.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Do fixed V-Rubrics scores establish actual visual judging?
V-Rubrics: Visual Faithfulness via Rubric-Based Reinforcement Learning
Show answer and example
Answer: No. Deterministic public candidates/plans test reward and credit without paper checkpoints, private data or full benchmarks. Fixed scores aren't vision.
Illustrative example: A supplied VF score tests aggregation, not seeing an image.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why create a reflection patch after SRPO failure?
SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning
Show answer and example
Answer: The patch converts coarse outcome feedback into privileged corrective context. The same model uses it to score actual student prefixes densely, not copy the failed trace.
Illustrative example: After a tool error, a parameter-check reminder can guide teacher rescoring.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why keep original rollout prefixes when a same-weight teacher sees a reflection patch?
SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning
Show answer and example
Answer: The patch is hindsight training context for evaluating visited states. Rewriting student prefixes or giving the patch at deployment changes the task. Token teacher/old-student log ratios supply dense credit; reflection isn't independently proven truth.
Illustrative example: After a failed retrieval, a teacher may know the missed constraint while scoring, but the student didn't know it during the original choice.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does SRPO self-reflection differ from an independent external teacher?
SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning
Show answer and example
Answer: Weights are shared and conditioning differs via feedback/patch, not additional parameters. A bad reflection can amplify shared bias; it isn't independent verification.
Illustrative example: Someone judging after seeing the outcome knows more, but isn't a second expert.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Should SRPO rescore a teacher-generated ideal prefix?
SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning
Show answer and example
Answer: No. Keep actual student on-policy prefixes; patch changes teacher context only. Detach reward difference and advantage. Local proxy axes do not generate real patches or execute a large tool model.
Illustrative example: Score the student's wrong third step as it occurred instead of replacing it with a correct one.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why track reward saturation in SA-MRPO?
Show answer and example
Answer: Already mastered objectives can crowd out unlearned dimensions. Standardize each dimension and weight by one minus saturation; the saturation estimate must be meaningful.
Illustrative example: If format nearly always passes, prioritize still-failing factual checks.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why adjust weights based on saturation instead of always treating objectives equally?
Show answer and example
Answer: Mastered dimensions can waste budget needed elsewhere. Normalize per dimension, then reduce saturated goals by a specified statistic. High scores don't mean permanent mastery, and raw scale isn't mastery.
Illustrative example: Reduce formatting pressure if already solved, but check whether later reasoning training degrades it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does saturation weighting differ from fixed reward weights?
Show answer and example
Answer: Fixed weights ignore current mastery; saturation weighting adapts by batch. It doesn't change ultimate importance or prevent forgetting.
Illustrative example: After downweighting a mastered skill, keep monitoring rather than declare it permanently solved.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Can SA-MRPO compute one saturation after aggregating rewards?
Show answer and example
Answer: No. That removes per-dimension budget allocation. Track each mean, variance and saturation before aggregation. Candidate formulas don't reproduce checkpoint training.
Illustrative example: One aggregate hides low correctness and high format performance.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why randomly hide rubric criteria?
Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL
Show answer and example
Answer: Fixed proxies can be repeatedly exploited. Random subsets make a single shortcut unreliable. Hidden criteria are not unimportant; the visible reward composition changes during training.
Illustrative example: A canned phrase cannot reliably earn total reward if its easy criterion is often absent.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why share a rubric subset within a rollout group?
Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL
Show answer and example
Answer: Relative rewards need common criteria; different random criteria can misrank peers. Cross-group randomization reduces fixed shortcuts, but doesn't guarantee valid remaining criteria or protect task performance.
Illustrative example: Omit brevity for both answers, not only one.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does group-shared dropout differ from independent masks per response?
Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL
Show answer and example
Answer: A shared subset keeps same-prompt responses comparable. Independent masks confound quality with scoring criteria. Randomize by group.
Illustrative example: Comparing A on format and B on correctness doesn't isolate answer quality.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
When should Rubric Dropout sample and save the mask?
Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL
Show answer and example
Answer: Sample one mask per rollout group, freeze for scoring and log seed. Never choose masks after seeing which response benefits. No local real rubric judge.
Illustrative example: Hiding difficult criteria only after observing failure is cherry-picking, not dropout.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why derive RRC generative reward from relative ranking?
Show answer and example
Answer: Generative judges may compare reliably without calibrated standalone scores. Within-group ranks plus anchors yield centered rewards while exploiting comparative strength.
Illustrative example: A judge may prefer one summary reliably but inconsistently use a ten-point scale.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why combine group and anchor ranking rather than ask for absolute scores?
Show answer and example
Answer: Comparative judgments can be more stable than absolute scale. Group competition gives relative position; anchors give cross-group reference before centering. This still depends on candidates and judge, not absolute truth or full paper evaluation.
Illustrative example: Two group winners may have very different quality relative to a shared anchor.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
What do group self-competition and anchor ranks each provide?
Show answer and example
Answer: Group comparisons separate current candidates; anchors bridge groups where ranks alone hide absolute level. Specify weights; rank gaps aren't equal quality distances.
Illustrative example: An all-bad group still has a winner; an anchor can show it remains poor.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Why check group size and rank direction before converting RRC ranks to rewards?
Show answer and example
Answer: K−1 requires more than one sample. Declare whether rank one is best. No local real judging or complete capability integration.
Illustrative example: A reversed formula can give the best rank the lowest reward.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why not simply trust high privileged-teacher confidence in ADRS?
Show answer and example
Answer: Extra skills/context may boost confidence without returns. Gate dense feedback by correlation between same-step teacher scores and observed returns.
Illustrative example: A teacher can favor a tool action that frequently fails.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why check teacher confidence against realized return before using token guidance?
Show answer and example
Answer: Privileged certainty can amplify error. Normalize within interaction steps, gate by positive return association and use the existing reward-to-advantage path. Negative association isn't help; inference does not use privileged training skill/outcome.
Illustrative example: Gate out confidence in fluent but ineffective tool calls.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does TVA correlation gating differ from fixed distillation weight?
Show answer and example
Answer: Fixed weight always trusts teacher; positive correlation retains signals associated with returns. Batch correlation is not causality and may not use test gold.
Illustrative example: If teacher scores don't track success, extra guidance weakens.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Does ADRS standardize over a full trajectory or within interaction steps?
Show answer and example
Answer: Standardize teacher scores within each interaction step and correlate across trajectory returns, then feed host reward-to-advantage logic. Local z-scores aren't correctness labels.
Illustrative example: Compare second tool calls with other second calls, not every turn combined.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why keep a Good–Normal–Bad archive in SERPO?
SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning
Show answer and example
Answer: Different qualities inform prompt-specific rubrics and policy learning. Actor and criteria evolve; storing only highest scores loses contrasts. Classification depends on judge quality.
Illustrative example: Medium answers expose gaps and bad answers show patterns to avoid.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why separate SERPO archive, rubric and actor roles?
SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning
Show answer and example
Answer: The archive stores comparisons, rubric defines current standards, and actor generates. Feedback can adapt or amplify bias. New archive or rubric doesn't itself train or improve the model; use independent holdout.
Illustrative example: A short-only archive, brevity rubric and short actor may agree while neglecting facts.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does SERPO differ from repeated RL under a fixed rubric?
SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning
Show answer and example
Answer: It updates query-specific rubrics and archives with the actor. Fixed criteria can lose discrimination, but coevolution risks drift and self-reinforcement. Self-scores aren't proof.
Illustrative example: New criteria can distinguish harder negatives after the actor learns initial rules.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
How should SERPO track its three evolving states?
SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning
Show answer and example
Answer: Record versions and ordering of generation, archive, rubric and actor updates. Don't retrospectively score with later rubrics. Local candidates don't implement full generation/judging.
Illustrative example: Log the actual rubric used for each batch so training supervision is auditable.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why replay one reply with and without criteria in CoRT?
Show answer and example
Answer: Keep response fixed so likelihood differences link rubric context to token positions, redistributing response advantage rather than adding another global score.
Illustrative example: A format requirement can affect bracket tokens and their credit.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why replay the same response instead of generating separately under each rubric view?
Show answer and example
Answer: Fixed tokens/prefixes isolate conditioning better; regenerating also changes trajectories. Use the difference to allocate original advantage, not pass scoring text into deployment. Counterfactual contexts can still bias the estimate.
Illustrative example: A citation token's likelihood can change with a citation requirement; regenerating the whole paragraph adds other changes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does CoRT counterfactual replay differ from attention weighting?
Show answer and example
Answer: Replay changes evaluation context and compares probabilities; attention measures one forward's information weights. Neither directly proves causality; attention isn't the specified contrast.
Illustrative example: A high-attention token may show unchanged likelihood when criteria are removed.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Why align both CoRT scoring passes to the same token and prefix?
Show answer and example
Answer: Different generated text confounds the criteria intervention. Normalized weights scale fixed advantage; original student sampling must not see the privileged rubric. Candidate features aren't token replay.
Illustrative example: Scoring sentence A with criteria and B without doesn't isolate their effect.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why subtract outcome EMA from TCR process scores?
Show answer and example
Answer: Process and outcome can double-reward success. The residual captures improvement not already explained by outcome before combining them. EMA is training state, not item truth.
Illustrative example: A correct answer should not automatically earn the same success signal again through process scores.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why subtract a moving outcome baseline rather than add both rewards directly?
Show answer and example
Answer: Process ratings often track correctness, so direct sums can count success twice. The moving residual attempts to isolate extra reasoning benefit. It depends on checklist and baseline quality, and local proxies don't establish full reasoning performance.
Illustrative example: If correct answers routinely earn high process scores, only an improvement beyond the usual outcome level gives new evidence.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does TCR residual differ from adding process and outcome rewards?
Show answer and example
Answer: Addition accumulates both signals. Subtracting a dynamic outcome baseline reduces duplicate credit. It does not prove exact decorrelation or remove judge bias.
Illustrative example: A higher outcome baseline reduces extra credit for an unchanged process score.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Does local TCR call a model to generate per-example checklists?
Show answer and example
Answer: No. Candidate policies test residual computation, not real checklist generation or free-text training. Update EMA in training order, not from test performance.
Illustrative example: A hand-provided checklist does not validate automatic checklist quality.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why build DynamicRubric criteria from the current candidate set?
Co-Evolving LLM Evaluators and Policies via DynamicRubric
Show answer and example
Answer: Old rules can saturate after policy gains. New hard negatives need discriminating dimensions and weights, while anchors limit judge drift. Not permission to change rules after outcomes arbitrarily.
Illustrative example: Once every answer meets formatting, look for evidence that still separates quality.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why balance hard-negative discrimination and anchoring?
Co-Evolving LLM Evaluators and Policies via DynamicRubric
Show answer and example
Answer: Fixed rubrics can stop separating improved replies; dynamic dimensions help. Discrimination alone can learn stylistic shortcuts, so an anchor stabilizes the reference. Coevolution is not objective improvement without independent tests.
Illustrative example: When formatting saturates, new criteria should not simply favor verbosity unchecked.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How do discrimination and anchoring pull DynamicRubric in different directions?
Co-Evolving LLM Evaluators and Policies via DynamicRubric
Show answer and example
Answer: Discrimination increases sensitivity to current hard cases; anchors retain prior standards and deter co-adaptation. Either alone can reward irrelevant differences or preserve stale criteria.
Illustrative example: Length is easy to distinguish, but an anchor should constrain it if unrelated to quality.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Why tie DynamicRubric weights to prompt and candidate version?
Co-Evolving LLM Evaluators and Policies via DynamicRubric
Show answer and example
Answer: Both determine scoring and new candidates may imply new criteria. Log rubric and anchor versions; don't compare rewards under undeclared weight changes.
Illustrative example: Score changes across two rounds can reflect changed criteria, not actor gains.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why can a correct TACO response contain tokens that shouldn't be reinforced?
Show answer and example
Answer: Terminal correctness only validates the result. Uniform positive credit can reinforce lucky or implausible intermediate tokens. Tail risk smoothly reduces their positive contribution.
Illustrative example: A correct computation with a false explanation is not wholly exemplary.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why reduce only positive high-tail-risk credit?
Show answer and example
Answer: Correct trajectories can contain unlikely, unsound steps. Reduce their positive reinforcement smoothly while retaining negative signals for failures. Contextual risk is not a low-probability equals wrong rule; no real local predictor.
Illustrative example: A lucky correct guess needn't be rewarded as strongly, but its negative feedback in a failed trajectory remains.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
Does TACO scale positive and negative advantages the same way?
Show answer and example
Answer: No. Reduce positive credit only for high-risk tokens, leaving nonpositive advantages unchanged. This preserves failure learning.
Illustrative example: A rare word in a failed response should not escape the original penalty solely for high risk.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Does local TACO reproduce a contextual tail-risk predictor?
Show answer and example
Answer: No. Candidate actions verify credit direction, not long-CoT token risk. Smooth weights must be between zero and one; don't simply delete all unlikely tokens.
Illustrative example: A supplied risk label tests branching, not automatic detection of implausibility.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why consider coalitional marginal contributions of spans in SCAR?
SCAR: Shapley Credit Assignment for More Efficient RLHF
Show answer and example
Answer: A span's value depends on other spans. Shapley weights gains over coalitions to allocate terminal reward, including negative contributions.
Illustrative example: An explanation can help only when its premise is also present.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why retain negative Shapley values and credit only the span's final token?
SCAR: Shapley Credit Assignment for More Efficient RLHF
Show answer and example
Answer: A span may reduce frozen RM value, so clipping breaks decomposition. End-token placement and original-order coalitions define reward scattering. Total equivalence has discount conditions; small exact enumeration isn't Owen approximation or full RLHF.
Illustrative example: A contradictory extra paragraph can reduce value; zeroing its negative credit breaks total equality.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does the exact local Shapley component differ from the paper's SHAP/Owen pipeline?
SCAR: Shapley Credit Assignment for More Efficient RLHF
Show answer and example
Answer: Enumeration is exponential and accepts small complete value tables only. The full method also has adaptive segmentation and hierarchical approximation.
Illustrative example: Three spans permit all coalitions; dozens of spans do not fit the same budget.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Why place SCAR credit at span end and preserve masked positions?
SCAR: Shapley Credit Assignment for More Efficient RLHF
Show answer and example
Answer: Score coalitions in original order, filling excluded positions with spaces, and assign credit to each span's last token. Keep negatives and zero empty value. Reward conservation requires the corresponding γ=1 condition.
Illustrative example: Joining text across removed middle spans changes the defined coalition.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Where does INTUITOR reward come from without an external verifier?
Show answer and example
Answer: Compute self-certainty from each prefix's full next-token distribution and average across the response. This is intrinsic signal, not correctness; use independent evaluation.
Illustrative example: A sharp distribution can still produce wrong arithmetic.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Can negative entropy replace INTUITOR self-certainty?
Show answer and example
Answer: No. The paper uses KL(U||p), unlike KL(p||U) or negative entropy; low-probability tails affect directions differently. Local sampled-candidate ratios aren't tokenwise reproduction. Confidence isn't correctness.
Illustrative example: Two distributions with similar maxima can have very different small tails, which KL directions weight differently.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
Can INTUITOR's intrinsic KL and reference KL be merged?
Show answer and example
Answer: No. Uniform-to-current KL measures certainty; reference KL anchors drift. They use different references and roles. Treating intrinsic reward as regularization changes the target.
Illustrative example: One encourages certainty while another discourages moving from old policy, potentially opposing each other.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Does current intuitor code implement full-vocabulary self-certainty?
Show answer and example
Answer: No. Reward uses sampled-candidate logp+log N and diagnostic KL(p||U), lacking token-level full-vocabulary KL(U||p). Historical scores remain diagnostics even if docs are corrected.
Illustrative example: Changing the documented direction does not alter the executable branch.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
How does Math-Shepherd get step supervision without human step labels?
Show answer and example
Answer: Continue repeatedly from a prefix and estimate reachable success from final answer checks, then train a process verifier. This is not direct human judgment of each line.
Illustrative example: A prefix with mostly correct continuations receives a higher success estimate.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why isn't a continuation-based step label a direct correctness judgment?
Show answer and example
Answer: It estimates success under a particular policy and sampling budget. Correct prefixes can get low estimates and wrong ones can be repaired. Separate reachability from correctness; use gold only for annotation or independent scoring.
Illustrative example: Low success after a correct second step may reflect frequent final arithmetic mistakes, not a wrong second step.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
How does continuation success differ from correctness of a step?
Show answer and example
Answer: Later policy can repair an error or fail after correct reasoning. Estimated reachability combines prefix and continuation policy, not a logical proof.
Illustrative example: A wrong step followed by frequent corrections should not be imitated simply because continuations succeed.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Can Math-Shepherd continuations see gold when constructing labels?
Show answer and example
Answer: No. Gold is only for terminal checking; actually sample from the specified prefix rather than constructing a path from known answers. No full local LLM data/PRM training.
Illustrative example: The grader can know the result without including it in the continuation prompt.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why do process supervision methods label steps rather than only outcomes?
Show answer and example
Answer: Failure doesn't identify the broken step. Local labels provide finer learning and selection, requiring annotations or reliable process judges. Do not broadcast terminal correctness as step truth.
Illustrative example: If step three fails, steps one and two needn't be marked wrong.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why actively label uncertain steps instead of only checking final answers?
Show answer and example
Answer: Final outcomes don't locate early mistakes. Step labels provide detailed reward-model supervision; uncertainty targets current ambiguous decisions but isn't a universal error proxy. Control candidate/evaluation budgets; no full local annotation dataset.
Illustrative example: Review an uncertain key algebraic move rather than repeatedly labeling obvious format.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
What tradeoff separates active uncertain-step labeling from random labeling?
Show answer and example
Answer: Active selection spends budget on hard judgments; random selection better reflects the original distribution. Uncertainty isn't importance and selection bias/label quality remain.
Illustrative example: An ambiguous formatting step can be less valuable to review than a key mathematical transition.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
How separate PRM labels and predictions?
Show answer and example
Answer: Step truth is fixed supervision, probabilities differentiable, and entropy used only in allowed acquisition. Test step labels cannot select training samples. Candidate budget tests mechanism only.
Illustrative example: Training on observed test-label errors leaks evaluation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Basics
Why can standardized rewards violate primary priority despite gating raw auxiliary rewards?
CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning
Show answer and example
Answer: The raw gate constrains auxiliary scores for failures. Independent centering/scaling can amplify small auxiliary gaps enough to reverse primary order. At a primary ceiling the primary advantage is zero, so auxiliary can determine the update. CONCUR constrains standardized terms, not just the raw gate.
Illustrative example: Example: Two correct answers differ slightly in length reward. Standardization can give length unit variance while primary advantage is zero; this is not evidence of learning more correct answers.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Core mechanism
Why sort by primary then auxiliary before PAVA without averaging all equal primary scores?
CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning
Show answer and example
Answer: Monotonicity only constrains strictly higher primary levels, not equal non-ceiling primary values. Sorting ties by auxiliary lets PAVA fix cross-primary violations only. Equal outputs are required at the ceiling only when that tie is enabled.
Illustrative example: Example: Primary [0.3,0.3,0.7], auxiliary [-1,0,1] is already feasible. Replacing the two first values with −0.5 adds a constraint.
Sections in the Chinese source: 核心公式, 本地结果与负控. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Method comparison
Does CONCUR-A exceeding two standard errors prove auxiliary reliability at the ceiling?
CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning
Show answer and example
Answer: No. It measures concordance of passing pairs with different primary values, releasing ties after at least 61 valid readings and mean greater than two standard errors. The evolving distribution is not IID, and ceiling pairs are not directly measured. This is not a significance level or universal guarantee.
Illustrative example: Example: Concordance can reverse later and drop below the threshold; rebind the ceiling rather than treating the first release as permanent.
Sections in the Chinese source: 核心公式, 数据协议与实现边界. The example illustrates the mechanism.
LLM post-training / Reward and credit: which steps deserve a good final score?
Implementation details
Can the first group in a batch decide the CONCUR-A ceiling from only previous history?
CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning
Show answer and example
Answer: No. Algorithm 1 merges every valid reading in the batch before sharing one policy per task/auxiliary and projecting groups. Sequential updates would depend on group order. Final whitening spans the full multi-task batch, not each group. A tensor check is not LLM lift.
Illustrative example: Example: History has 60 valid positives and group two supplies the 61st; group one also uses the merged current trust state.
Sections in the Chinese source: 核心公式, 数据协议与实现边界, 代码映射. The example illustrates the mechanism.
Topic guide and reading order
Start with this question¶
How can one final score become a training signal that does not reward wrong steps?
Takeaway¶
Reward defines what is good; credit determines which actions receive it. Process supervision labels intermediate steps. Math-Shepherd samples continuations and uses their eventual solvability to construct step labels. Rubrics and multiple objectives must map criteria to tokens and prevent one reward from overpowering another. Outcome rewards, process evidence, and advantages are related but distinct. CONCUR shows that even correctness-gated length rewards can reverse primary-reward order after normalization, motivating an explicit projection constraint rather than only reducing the auxiliary weight.
Example¶
A math response can reason well for two steps, make a mistake at step three, and still guess the correct answer. Applying terminal reward to all steps can reinforce the error. Independent continuations or step checks give finer evidence, but a failed continuation may reflect inadequate sampling rather than an incorrect prefix.
Common pitfall¶
Judge agreement does not establish truth, and style or shortcuts can manipulate rubric scores. Evaluate label quality, the credit formula, and capability separately. Keep hidden answers inside the evaluator. Synthetic credit checks are not evidence of reasoning, robotics, or video-generation ability.
Reading order¶
Read Process Supervision for step versus outcome labels, Math-Shepherd for continuation-based labels, and TACO for why even correct answers can contain tokens that should not be reinforced.
Check your understanding¶
If a long answer ends correctly, can uniform positive token credit prove every step was correct?
Show suggested answer
No. Terminal reward says only that the final output met the acceptance condition. Uniform credit is an estimation choice that can reinforce wrong steps. Fine-grained credit likewise needs a reliable source and independent comparisons; a judge score is not process ground truth.