Trajectory distillation: students, proposals, and prefix distributions¶
简体中文 · English
LLM post-training / 168 questions · 42 papers
Try answering each question before opening the explanation. The bank grows as papers and questions are added.
Read this topic's comparisons, analysis, and cross-paper Q&A
168 questions · 42 papers
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why is SpecOPD on-policy if its reference remains fixed?
SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding
Show answer and example
Answer: Only the outer reference context is fixed. Each update samples new branches from the current draft after anchors, so teacher-supervised prefixes contain the draft's own history. The complete reference response need not be sampled by the current policy, and collection need not wait for sequential draft-and-verify rounds to finish a response.
Illustrative example: Teaching example: The reference has quiet after a, but the current draft samples short. The next teacher query must condition on a short, not a quiet; these are different histories.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why does reordering identical token-overlap values change the SpecOPD loss?
SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding
Show answer and example
Answer: E2E-TV sums overlap products from the beginning of each branch to every position. Early mismatch reduces every subsequent prefix term; a final mismatch affects only the final term. It is not mean token TV. For history-dependent proposals it is a training surrogate, not an unbiased serving-acceptance estimate.
Illustrative example: Teaching example: Overlaps [0.6,1,1] produce prefix values 0.6,0.6,0.6; [1,1,0.6] gives 1,1,0.6 and a lower loss. These are illustrative, not model measurements.
Sections in the Chinese source: 核心公式, 结果、负控与边界. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
When the draft uses top-k support, how does renormalizing the teacher there change the comparison?
SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding
Show answer and example
Answer: It discards teacher probability on tokens the draft cannot propose, changing the acceptance-overlap objective. SpecOPD retains full-vocabulary teacher normalization. Outside draft support, q is zero but the missing teacher mass still matters. Replacing the teacher by a top-k conditional distribution is not an innocent computational shortcut.
Illustrative example: Teaching example: q=[.5,.5,0] and p=[.4,.4,.2] have overlap .8. Renormalizing the first two teacher entries invents a perfect match, incorrectly changing loss .2 to zero.
Sections in the Chinese source: 核心公式, 结果、负控与边界. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Why check prediction rows separately when the proposal tree already has a causal mask?
SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding
Show answer and example
Answer: Ancestry masking blocks future reference tokens and siblings. But predicting a sampled token from its own hidden-state row already exposes the answer as input. The first token must read its anchor and later tokens their predecessors, with semantic rather than packed positions. Correct masking does not replace correct readout.
Illustrative example: Teaching example: For samples short evening, the anchor predicts short and the short row predicts evening. Scoring evening from its own row still leaks it even if that row sees only its ancestors.
Sections in the Chinese source: 核心公式, 数据协议与代码映射, 结果、负控与边界. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why does OPST generate both problems and privileged clues?
On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models
Show answer and example
Answer: Reduce reliance on human problem banks and give teacher extra solving context while students generate from prompts only. Validate generated tasks through independent re-answering, meaning-preserving rewrites and difficulty filtering.
Illustrative example: Resolve a generated problem independently to reject inconsistent prompts or answers.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why reject as easy only unanimous matches to the verified candidate answer?
On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models
Show answer and example
Answer: Agreement alone doesn't show mastery. All students can agree on the wrong answer. Privileged clues only guide teacher scoring on student prefixes; self-generated problems need independent re-answering and meaning-preserving checks.
Illustrative example: A unanimous wrong number remains useful until compared with the verified candidate answer.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does OPST easy filtering differ from removing student consensus?
On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models
Show answer and example
Answer: Only unanimous matches to the candidate answer are too easy. Unanimous disagreement may signal a gap or bad problem. Removing all consensus changes the gate.
Illustrative example: Identical wrong results don't establish mastery.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Is OPST EMA β old-teacher momentum or new-student weight?
On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models
Show answer and example
Answer: It is new-student weight: after each student update, use (1−β) old teacher + β new student. Normalize per response then average equally. Document formula ambiguity instead of claiming unquestioned exactness.
Illustrative example: Small β means slowly absorb the student, not almost replace the teacher.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why does UserArena report more than next-action accuracy?
UserArena: Benchmarking Interactive User Simulation for Long-Horizon Shopping Agents
Show answer and example
Answer: Simulation should preserve trajectory dynamics, cross-session preferences and cohort differences, and reproduce differences between recommendation policies. Local hits aren't whole-user fidelity.
Illustrative example: Good click prediction with wrong long-run interest drift can still distort policy evaluation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why give only teacher abstract intent and bound it token by token?
UserArena: Benchmarking Interactive User Simulation for Long-Horizon Shopping Agents
Show answer and example
Answer: Whole trajectories leak future products/actions. Filter abstract preferences for teacher only, preserving student deployment input. Compare teacher views on the same prefix, constrain added guidance and ground on actual responses. KL can't replace leakage checks.
Illustrative example: Durability preference may be available; next item IDs and purchase outcomes may not.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does OPTD teacher intent differ from giving the student future trajectories?
UserArena: Benchmarking Interactive User Simulation for Long-Horizon Shopping Agents
Show answer and example
Answer: Only frozen teacher sees abstract intent, excluding concrete action/item identity, price and outcome. Student continues its own prefixes. Trust regions bound effects but don't prove no extraction leakage.
Illustrative example: Lightweight-item preference is allowed, next clicked product is not.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
May OPTD grounding replace gold tokens on a student rollout prefix?
UserArena: Benchmarking Interactive User Simulation for Long-Horizon Shopping Agents
Show answer and example
Answer: No. Grounding uses separate true-prefix logits; teacher views score identical student prefixes. Use feasible lower endpoint per token with KL from mixed to no-intent teacher, not batch-mean reverse KL.
Illustrative example: Injecting the actual next action after a divergent student history is not teacher forcing on a real trajectory.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why inspect RL and distillation gradient geometry in GIFT?
GIFT: Gradient Informed Filtering for Distillation Augmented Reasoning RL
Show answer and example
Answer: Teacher signal can conflict or add an orthogonal direction. Select by local logit gradients to reduce conflict and narrow supervision rather than imitate every token.
Illustrative example: Teacher format preferences can oppose task reward, so inspect the directions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why can't gradient-based GIFT selection promise permanent absence of conflict?
GIFT: Gradient Informed Filtering for Distillation Augmented Reasoning RL
Show answer and example
Answer: Prioritize nonconflicting and orthogonally informative tokens; quotas can require weaker conflicts. Selection isn't projection, and normalize by all valid tokens. Local logit geometry doesn't guarantee finite shared-parameter updates.
Illustrative example: A subbatch quota larger than conflict-free availability can select the least conflicting extras.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does GIFT token selection differ from projecting teacher gradients?
GIFT: Gradient Informed Filtering for Distillation Augmented Reasoning RL
Show answer and example
Answer: Selected tokens retain original distillation losses; gradients aren't rotated. Exact quotas may admit weak conflicts, so there is no universal finite-step safety guarantee.
Illustrative example: An accepted position uses its original loss, not a geometrically replaced gradient.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Can GIFT renormalize by selected token count?
GIFT: Gradient Informed Filtering for Distillation Augmented Reasoning RL
Show answer and example
Answer: No. Divide by all valid tokens. Quotas depend on subbatch valid/eligible sets. Follow explicit handling for missing RL and zero teacher gradients rather than automatically treating them as novel directions.
Illustrative example: Selecting half and dividing by half doubles retained contribution.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why minimally repair a wrong student response before projected distillation?
Projected Distillation: On Learning to Solve Hard Problems from Close, Correct Targets
Show answer and example
Answer: Expert prose can be far from student distribution. Minimal edits retain current behavior while correcting what is needed, with independent verification. This is an approximation, not exact most-likely correct output.
Illustrative example: Correct one wrong symbol instead of replacing the whole derivation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why restrict DPO to edits and SFT to student successes?
Projected Distillation: On Learning to Solve Hard Problems from Close, Correct Targets
Show answer and example
Answer: Minimal repair preserves generated structure and focuses preference credit on corrections. Repaired and independently correct responses have different provenance. Score both sides under their full causal contexts; shared parameters can still change.
Illustrative example: If only the last operator is wrong, verify its replacement and contrast the edit rather than imitate all teacher wording.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does edited-span DPO differ from SFT on a complete repaired answer?
Projected Distillation: On Learning to Solve Hard Problems from Close, Correct Targets
Show answer and example
Answer: Span DPO contrasts only original and verified edit positions; unchanged text is not directly included. Full SFT is for student-generated successes, not projector repairs.
Illustrative example: An externally repaired reply is not an autonomous success from this round.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Can edited-span scoring drop the preceding text?
Projected Distillation: On Learning to Solve Hard Problems from Close, Correct Targets
Show answer and example
Answer: No. Score each edit under its own complete causal prefix, summing span logp without length averaging. One insertion/deletion side may be empty but pair needs an edit. Zero direct gradient elsewhere does not protect shared weights.
Illustrative example: Editing step three changes step four's context; don't borrow another response's prefix.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why doesn't inter-layer OPTD only align final predictions?
On-Policy Transition Distillation
Show answer and example
Answer: Final agreement can hide different depth-wise revisions. Supervise shallow-to-deep log-probability increments and retain final OPD. This is prediction paths, not hidden-vector matching.
Illustrative example: Two models can end on A after very different intermediate changes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why choose candidate tokens once at the final student layer?
On-Policy Transition Distillation
Show answer and example
Answer: Track the same predictions through depth. Independent top-k confounds identity with changes. Use full-vocabulary-normalized log-probability increments plus final OPD, with detach and destination-layer weights, not hidden MSE or exact transition KL.
Illustrative example: If shallow favors A and deep B, retain both identities to trace their rise/fall.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does transition supervision differ from independent top-k at each layer?
On-Policy Transition Distillation
Show answer and example
Answer: Final student layer chooses fixed token identities for the entire path. Independent sets mix different objects when comparing increments.
Illustrative example: If shallow top is A and deep top B, subtracting their logp doesn't measure A's change.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Which OPTD increment endpoints receive gradients and how is practical scale handled?
On-Policy Transition Distillation
Show answer and example
Answer: Both adjacent student readouts are differentiable. Teacher, candidate weights and correction coefficients are frozen. Appendix sums transitions while main text averages; neither is exact reverse transition KL.
Illustrative example: A sum of three transitions cannot silently become a mean at the same objective scale.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why does AP-OPD target penalty jumps rather than uniformly high penalties?
Adaptive Pullback: Reweighting Supervision at Penalty Jumps in On-Policy Distillation
Show answer and example
Answer: Persistent disagreement and sudden local deterioration differ. Compare adjacent original block statistics and attenuate the later block to limit spike dominance.
Illustrative example: A hard entire response differs from one getting abruptly worse after a particular step.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why use original block statistics and scale negative coefficients too?
Adaptive Pullback: Reweighting Supervision at Penalty Jumps in On-Policy Distillation
Show answer and example
Answer: Weighted statistics would recursively alter subsequent reference levels. Detect from original positive penalty means but attenuate all signed supervision in the later block. Preserve direction, allow zero weight, and do not interpret threshold as calibrated significance.
Illustrative example: Compare block three to block two's original level even if block two was already reduced.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does AP-OPD attenuation differ from dropping outlier tokens?
Adaptive Pullback: Reweighting Supervision at Penalty Jumps in On-Policy Distillation
Show answer and example
Answer: Apply one factor to all signed coefficients in a block, preserving signs and including negatives. Dropping tokens is a different selection rule. The detector is not a calibrated significance test.
Illustrative example: Negative log ratios in the later block are also scaled.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Can the next AP-OPD boundary use previously attenuated statistics?
Adaptive Pullback: Reweighting Supervision at Penalty Jumps in On-Policy Distillation
Show answer and example
Answer: No. Always detect from original unweighted means/variance to avoid recursive distortion. No renormalization or hidden floor; previous mean zero may produce zero weight. Sample variance needs enough tokens.
Illustrative example: Detecting from an already reduced block can create artificial jumps.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why can SimOPD handle different teacher/student hidden dimensions?
SimOPD: Relational Knowledge Transfer for On-Policy Distillation
Show answer and example
Answer: Each space compares current reply to paired historical replies. Distill distributions indexed by shared answer identities rather than subtract cross-model vectors.
Illustrative example: A 100-dimensional model and wider model can both rank similarities to the same old answers.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why not project teacher/student hidden states into one coordinate system first?
SimOPD: Relational Knowledge Transfer for On-Policy Distillation
Show answer and example
Answer: Compute similarities in each own space, match distributions over paired historical IDs. Freeze teacher/bank and maintain synchronized FIFO. A single historical point cannot provide relational supervision.
Illustrative example: Two teachers can agree a reply is closer to historical A than B without identical coordinates.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does relational SimOPD differ from hidden-state MSE?
SimOPD: Relational Knowledge Transfer for On-Policy Distillation
Show answer and example
Answer: MSE requires comparable coordinates. Relationship matching preserves geometry without forcing coordinates and differs from vocabulary distillation: targets are similarities to historical answers.
Illustrative example: Rotating a map changes coordinates without changing city distances.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
When does SimOPD enqueue and why reject a one-item bank?
SimOPD: Relational Knowledge Transfer for On-Policy Distillation
Show answer and example
Answer: After optimization enqueue frozen paired representations captured before it, using synchronized FIFO; first batch only warms up. One bank item makes the distribution constant and uninformative. Only current student representation gets gradients.
Illustrative example: Misaligned FIFO positions compare relations to different answers.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why pool token feedback into sequence scores in GR-OPD?
GR-OPD: Group-Relative Sequence Credit for On-Policy Distillation
Show answer and example
Answer: Local teacher confidence needn't mean good overall reasoning. Discount and aggregate sequence feedback, then compare same-prompt replies for relative credit.
Illustrative example: Teacher preference for formatting tokens doesn't make a wrong trajectory preferable.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why retain mean pressure after group centering?
GR-OPD: Group-Relative Sequence Credit for On-Policy Distillation
Show answer and example
Answer: Pure centering keeps only within-group ranking and can erase shared correction. Add mean pressure before broadcasting credit to actions. Discount, standardization and clipping alter the practical estimator's guarantees.
Illustrative example: All replies can be similarly unsupported; centering nearly zeroes them while mean term preserves correction.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does retaining mean pressure differ from pure group centering?
GR-OPD: Group-Relative Sequence Credit for On-Policy Distillation
Show answer and example
Answer: Pure centering removes teacher's common direction across replies. An extra mean term preserves some shared pressure; its coefficient may be zero. Treat configurations separately.
Illustrative example: If every reply diverges from teacher, relative ranks alone hide shared mismatch.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Does GR-OPD average tokens globally or per response?
GR-OPD: Group-Relative Sequence Credit for On-Policy Distillation
Show answer and example
Answer: Average valid actor tokens within each response then average replies equally. Clip and discount-normalize feedback first. Freeze teacher/old policy/credit. Bare theoretical sums don't imply guarantees under practical clipping and reuse.
Illustrative example: Length alone doesn't give a response more total weight.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why judge IG-OPD queries after retrieval?
Information-Gain Calibrated On-Policy Distillation for Deep Search Agents
Show answer and example
Answer: Teacher phrasing preferences can dislike effective searches. Changes in answer support after retrieval indicate useful information. Answer probes are training-only.
Illustrative example: An awkward query retrieving key evidence can be valuable despite its wording.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why budget with absolute gain but update queries with signed gain?
Information-Gain Calibrated On-Policy Distillation for Deep Search Agents
Show answer and example
Answer: Large positive or negative changes deserve supervision; action credit must distinguish help from harm. The additive parts act on different sets. Answers belong to frozen training probes only; long final replies must not absorb short-query budgets.
Illustrative example: Misleading results can justify large correction but negative rather than positive direction.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
Do ITC and IQC change credit magnitude or direction?
Information-Gain Calibrated On-Policy Distillation for Deep Search Agents
Show answer and example
Answer: ITC budgets turns by absolute gain; IQC uses signed gain to calibrate action tokens. Large negative gain merits attention, not positive reinforcement.
Illustrative example: Misleading retrieval that lowers confidence needs correction, not reward.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Why is IQC outside ITC, and what does λ=0 mean?
Information-Gain Calibrated On-Policy Distillation for Deep Search Agents
Show answer and example
Answer: Total action credit is warm-up coefficient times gain, not multiplied again by turn budget. λ=0 gives equal turn budget, not equal token weight. Final response gain is zero but base distillation remains.
Illustrative example: A short call and long answer with equal budgets have different token weights.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why add forward KL at high-disagreement positions in DrOPD?
Divergence-Routed On-Policy Distillation for Language Models
Show answer and example
Answer: Reverse-KL gradients depend on student mass and can be weak on teacher modes students nearly ignore. Forward KL restores missing coverage locally while retaining RKL everywhere.
Illustrative example: Teacher-supported digits with near-zero student mass need an extra coverage signal.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why keep RKL everywhere and add FKL only at high disagreement?
Divergence-Routed On-Policy Distillation for Language Models
Show answer and example
Answer: RKL imitates covered modes but can weakly move near-absent teacher modes. JS routing localizes FKL's coverage correction. Union support, detached advantages and PPO matter; not switching all KL or only updating the sampled token.
Illustrative example: If teacher strongly prefers A and student only B, extra local FKL can lift A.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does DrOPD routing differ from replacing the whole loss with forward KL?
Divergence-Routed On-Policy Distillation for Language Models
Show answer and example
Answer: Add FKL only on routed positions and keep RKL globally. Full replacement changes concentration/coverage elsewhere. PPO candidate-advantage updates also aren't a static two-KL gradient.
Illustrative example: Low-disagreement format tokens keep the original objective; difficult positions get extra correction.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Should ties at DrOPD's kth-largest JS be broken to keep exactly K?
Divergence-Routed On-Policy Distillation for Language Models
Show answer and example
Answer: No. Retain all at threshold, possibly more than K. Compute K from valid positions as specified; K=0 adds no FKL. Deduplicate and renormalize the top-k teacher/student union. Freeze teacher, route and advantage.
Illustrative example: Don't drop tied boundary positions to force an exact quota.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why hide old reasoning from the State-OPD teacher?
State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents
Show answer and example
Answer: Old reasoning can be wrong. The teacher sees executed actions/observations to avoid inheriting false narratives; student still sees its full history.
Illustrative example: Teacher should trust a tool's failure result over previous reasoning claiming success.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why omit old reasoning for the teacher and not execute extra actions?
State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents
Show answer and example
Answer: Executed observations are real even when student reasoning is mistaken. Use them for teacher state while student retains full context. Compare oversampled candidates at the same high-entropy state; executing them breaks alignment. Entropy is vocabulary-wide.
Illustrative example: Use actual click results instead of mistaken prior narrative, and score extra proposed clicks without pressing every button.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does high-entropy oversampling differ from executing all extra actions?
State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents
Show answer and example
Answer: Generate extra reasoning/actions for training comparison, but execute none of them. The actual environment follows the original trace. Executing them changes state visitation and budget.
Illustrative example: Scoring alternative tools doesn't mean using every one.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Can sampled-token surprisal replace State-OPD entropy?
State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents
Show answer and example
Answer: No. Average full-vocabulary entropies across tokens and follow the stated state selection/count formula. Keep sampled-K1 double clipping rather than heuristic teacher ranking.
Illustrative example: A sampled rare word does not imply high entropy of the whole distribution.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why fix trajectories generated by the initial student in Semi-OPD?
When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better
Show answer and example
Answer: One collection and teacher-scoring pass saves future costs. Train on fixed visited states, while acknowledging later policy drift means data are no longer freshly on-policy.
Illustrative example: Reuse initially common error prefixes without collecting newer ones.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why recompute student probabilities and advantages despite cached teacher scores?
When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better
Show answer and example
Answer: Fixed trajectories and teacher outputs can be reused, but student probabilities and gaps change after updates. Stale directions distort the objective. Offline student rollouts save compute, not fresh on-policy sampling every step.
Illustrative example: A once-underweighted token may exceed teacher support; old positive credit would still push it up.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
Why cache teacher labels but not advantages?
When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better
Show answer and example
Answer: Frozen teacher scores remain valid for fixed prefixes. Advantage also depends on current student logp and must refresh. A cached advantage is a stale objective.
Illustrative example: When student support rises, the teacher gap shrinks despite unchanged teacher score.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
What must remain immutable in the Semi-OPD cache?
When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better
Show answer and example
Answer: Align prompts, responses, teacher scores and masks. Refresh only current student scores and detached advantages each step. This doesn't guarantee long-term coverage.
Illustrative example: Deleting a token while reusing old indexed scores misaligns supervision.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why inspect both loss and gradients in OPD Learning Signals?
Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals
Show answer and example
Answer: Large residual divergence can coexist with weak updates. Combine loss, pre-clip gradients, parameter motion and representation similarity to diagnose stagnation.
Illustrative example: Distributions can differ while current parameterization yields little usable gradient.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why doesn't high residual OPD loss guarantee learnability?
Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals
Show answer and example
Answer: A numeric gap differs from gradient that can move parameters; saturation/coverage may weaken it. Jointly inspect pre-clip gradients, split estimates, parameter movement and representations. Nonpositive denominators are undefined. Diagnostics aren't causes.
Illustrative example: Loss can remain high while gradients and parameter updates are nearly zero.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
Do occupancy and CKA diagnostics prove why training stalled?
Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals
Show answer and example
Answer: No. They describe patterns among sampling, updates and representations. Causal attribution needs controls or assumptions.
Illustrative example: Similar layers and slow learning occurring together don't establish one caused the other.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
What if a learning-signal proxy has nonpositive loss or split denominator?
Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals
Show answer and example
Answer: Mark the ratio undefined, not zero-filled. Compute CKA from per-prompt averages centered across prompts at specified precision. Use gradients before clipping.
Illustrative example: A negative split gradient dot product should not become a positive occupancy by absolute value.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why provide positive and negative privileged contexts for each SIPO rollout?
SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
Show answer and example
Answer: Teacher logp differences aim to cancel common confidence/style and expose informative token evidence. Environment reward still guides the task. Student sampling sees neither privilege.
Illustrative example: If both teachers favor a filler phrase, subtraction reduces that common bias.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why use two SIPO teacher views rather than one correct hint?
SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
Show answer and example
Answer: Contrast log probabilities on identical student rollouts to reduce shared confidence and style. Environmental reward supplies main direction and dense evidence redistributes token credit. Both views are training-only.
Illustrative example: Both teachers may prefer politeness but diverge on key evidence; contrast downweights the shared style.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
What assumption differs between SIPO contrast and one privileged teacher?
SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
Show answer and example
Answer: Single-teacher confidence can include irrelevant effects. Contrast aims to isolate conditioning differences, but poorly designed contexts still mislead and are not independent truth.
Illustrative example: Swapping irrelevant wording needn't produce a useful solving contrast.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Can SIPO have token signal when all group rewards are equal?
SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation
Show answer and example
Answer: Yes. Centered contrast evidence can be nonzero with zero outcome advantage. Gold stays in teacher/evaluator only, not rollout.
Illustrative example: A failed group can receive directional teacher evidence without becoming autonomous successes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Does RIDE extrapolate output probabilities or hidden residuals?
Show answer and example
Answer: It compares RL teacher with pre-RL base per layer/token, then regresses the student to base plus amplified residual. This is representational change, not logit ratios.
Illustrative example: Extend a learned layer direction rather than multiply favored word probability.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why isn't hidden-residual extrapolation equivalent to increasing output probability ratios?
Show answer and example
Answer: The target is per-layer/token base plus scaled RL-teacher shift, an internal path probabilities do not preserve. Freeze teacher/base. α above one does not ensure capability beyond teacher.
Illustrative example: Extend RL's layer displacement rather than amplify all teacher-preferred output words.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
Does representation extrapolation with α>1 guarantee outperforming the teacher?
Show answer and example
Answer: No. Extending the direction may magnify benefits or move outside reliable states. Evaluate independently; farther geometry is not higher ability.
Illustrative example: Overshooting can destroy information despite extending the same direction.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Why stop gradients through RIDE teacher and base?
Show answer and example
Answer: The target is a fixed RL change; updating its branches makes residual drift. Align layers, tokens and dimensions before subtracting.
Illustrative example: Student error must not move the teacher target rather than update the student.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why separate OASIS scaffold and teacher context?
Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
Show answer and example
Answer: Use the shortest verified trajectory for supervision locations, preferably another unverified attempt for teacher context. Where to learn and what extra context to use are independent.
Illustrative example: A concise success can anchor positions while a longer attempt provides alternative clues.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why can an unverified attempt be teacher context but not scaffold?
Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
Show answer and example
Answer: A scaffold fixes a credible student trajectory; teacher context only provides comparison clues. Unverified attempts aren't correct labels. No verified rollout means no training signal, and final verification doesn't reveal gold to students.
Illustrative example: Score prefixes of a concise verified response while allowing the teacher to reference a longer exploration.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
Why require a verifier if OASIS has no gold solution?
Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
Show answer and example
Answer: It needs a verified successful rollout before selecting a scaffold. Absence of gold steps does not remove correctness checks. With none verified, emit no signal.
Illustrative example: The shortest unchecked answer is not automatically a valid scaffold.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Should OASIS order by length before or after checking eligibility?
Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
Show answer and example
Answer: Verify first, then choose the shortest eligible trajectory. Track scaffold and teacher context separately and hide future answers from student.
Illustrative example: A one-line wrong result must not beat a longer verified solution.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why interpolate IPD teacher/student sampling distributions?
Show answer and example
Answer: Pure student and teacher states trade coverage differently. Mix conditional policies continuously at each state; γ controls sample source distribution, not merely loss weighting.
Illustrative example: Let some steps follow student tendencies and other steps gain teacher-supported options.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why isn't mixing each token probability the same as selecting one model for the whole reply?
Show answer and example
Answer: Interpolate at every updated prefix; sampled tokens change later states. Choosing a model once gives a different sequence law, and averaging logits is also different. Endpoint checks do not validate the paper's speculative sampler.
Illustrative example: Sample a teacher-favored token first, then remix under the new prefix rather than fix a source for the rest.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
Is token-level mixing equivalent to choosing one model per rollout?
Show answer and example
Answer: No. Tokenwise mixtures can combine both preferences within a sequence; one source per rollout produces a different joint state distribution.
Illustrative example: Student-like first step and teacher-like second may be reached differently under fixed rollout source.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Do local IPD endpoint checks include speculative high-throughput sampling?
Show answer and example
Answer: No. Exact token mixture checks endpoints, normalization and total variation. γ=0 student, γ=1 teacher. Verification-aware speculative sampler and full training remain unreproduced.
Illustrative example: Calculating a mixture vector doesn't establish paper sampling throughput.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why separate rollout distribution and optimization kernel in FlowMap-OPD?
FlowMap-OPD: Rollout-Kernel Separation for On-Policy Distillation of Few-Step Flow-Map Generators
Show answer and example
Answer: Sampling supplies states with specified marginals; a local kernel compares teacher/student flows or velocities. Separate data provision from gradients rather than backpropagating through sampling arbitrarily.
Illustrative example: Freeze a sampled intermediate state, then compare local predictions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why freeze rollout states for kernel gradients?
FlowMap-OPD: Rollout-Kernel Separation for On-Policy Distillation of Few-Step Flow-Map Generators
Show answer and example
Answer: Rollout supplies required marginals; kernel compares flows or velocity at those states. Differentiating through generation adds unspecified trajectory terms. Detached states can be recollected from later policies, not permanently fixed.
Illustrative example: Sample intermediate states, freeze them for one local teacher comparison, and don't backpropagate through entire sampling.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does detaching states differ from end-to-end differentiation through the sampler?
FlowMap-OPD: Rollout-Kernel Separation for On-Policy Distillation of Few-Step Flow-Map Generators
Show answer and example
Answer: Detach trains student kernel parameters only. Differentiating through states adds dependency gradients and changes the estimator despite equal scalar loss.
Illustrative example: Current model may affect both states and predictions; freezing states deliberately cuts the former path.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Does local FlowMap train a real image generator?
FlowMap-OPD: Rollout-Kernel Separation for On-Policy Distillation of Few-Step Flow-Map Generators
Show answer and example
Answer: No. It verifies state detach and local KL. Freeze teacher; caller ensures required rollout marginals. Random vectors need not follow the paper's distribution.
Illustrative example: Computing KL for an arbitrary vector doesn't validate generation provenance.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why add future potential beyond current chunk distillation in DriftOPD?
DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies
Show answer and example
Answer: Actions alter future robot states. One-step matching misses downstream effects. Sequential reverse KL includes a future term estimated from offline demonstration critic.
Illustrative example: A plausible grasp can prevent later placement.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why add future potential to current chunk reverse KL?
DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies
Show answer and example
Answer: Immediate matching cannot capture state transitions. A demonstration-trained critic approximates long-horizon effects to retain timing in sequential decomposition. Coverage/critic accuracy matter; no local environment validation.
Illustrative example: Two grabs look teacher-like now, but one puts the object out of reach later.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does future potential differ from an immediate reward?
DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies
Show answer and example
Answer: It estimates downstream outcomes within a sequence decomposition; immediate reward rates current behavior. Critic approximation matters and adding value is not exact sequence KL.
Illustrative example: Lifting the object earns immediate credit even if placement becomes impossible.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Does local DriftOPD execute a robotics closed loop?
DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies
Show answer and example
Answer: No. It verifies objective decomposition and teacher detach. Real work needs an offline-trained future critic, chunk/state alignment and environment evaluation.
Illustrative example: A computed potential term does not show a robot completed the task.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why does Dr. OPD give teacher tokens different weights?
Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models
Show answer and example
Answer: Fixing key reasoning matters more than rephrasing. A bilevel approximation weights by expected post-update student reward, not just disagreement.
Illustrative example: At equal probability gap, correcting an operator may matter more than changing a connective.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why weight by downstream reward JVP rather than disagreement size alone?
Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models
Show answer and example
Answer: Large gaps may be stylistic. JVP estimates reward sensitivity along correction, then closed-form weights are bounded and normalized for OPD. Local reward and approximation do not prove global causal contribution.
Illustrative example: Teacher and student can disagree on an operator and politeness, but only one affects task reward.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does reward JVP weighting differ from ranking by KL disagreement?
Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models
Show answer and example
Answer: Disagreement measures distance; JVP estimates downstream reward change along a direction. Combine gap and value, but a local gradient isn't independent correctness evidence.
Illustrative example: A large stylistic disagreement can have weak reward effect.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
What order should Dr. OPD use for weights and student updates?
Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models
Show answer and example
Answer: Solve weights from fixed batch signals, clip and mean-normalize, then update student by weighted RKL. Do not differentiate the student through weight selection. Local d×g is a proxy.
Illustrative example: Choose supervision positions before the student step rather than changing the selector with that step.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why treat teacher–student logp gaps as robust residuals in LSPD?
An RL View of OPD: Least Square Policy Distillation for Sample-Efficient LLM Reasoning
Show answer and example
Answer: Squared loss shrinks ordinary gaps; Huber-like linear tails limit gradients from extreme ones. Residual regression can reuse older trajectories.
Illustrative example: A rare token's huge log gap should not dominate a batch indefinitely.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why use a quadratic center with Huber tails instead of squared loss throughout?
An RL View of OPD: Least Square Policy Distillation for Sample-Efficient LLM Reasoning
Show answer and example
Answer: Quadratic shrinks small logp gaps while linear tails bound outlier influence and retain correction direction. Average responses equally. Reused old traces are not fresh current-policy samples.
Illustrative example: One teacher-improbable token can dominate squared loss; robust tails curb it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does robust LSPD behave in tails compared with reverse KL?
An RL View of OPD: Least Square Policy Distillation for Sample-Efficient LLM Reasoning
Show answer and example
Answer: It explicitly limits residual slope; KL is defined through distribution/probability weights. Clipping KL isn't equivalent. Replay also changes state provenance.
Illustrative example: Doubling tail residual changes linear and unbounded quadratic gradients differently.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
How handle teacher gradients and lengths in LSPD?
An RL View of OPD: Least Square Policy Distillation for Sample-Efficient LLM Reasoning
Show answer and example
Answer: Freeze teacher logp. Average differentiable student gaps over valid tokens per response, then responses equally, with specified entropy subtraction. Registry presence doesn't prove executable Evolve.
Illustrative example: Long replies do not automatically receive twice the weight.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
How does Recursive OPSD DCE keep teacher aligned to student updates?
Recursive Self-Improvement via On-Policy Distillation for Reasoning
Show answer and example
Answer: Both use current shared weights, teacher additionally sees the reference solution and is detached. Updated weights become both views next round, not a separately trained or initial teacher.
Illustrative example: A newly learned correction is available to the next privileged teacher.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why impose different gold boundaries on DCE and SRCL although both update one model?
Recursive Self-Improvement via On-Policy Distillation for Reasoning
Show answer and example
Answer: DCE uses current privileged teacher to score actual prefixes, then refreshes both roles after updates. SRCL rewrites its own answer without gold and validates afterward. Giving gold to rewriter changes self-revision into copying. No second teacher is independently trained.
Illustrative example: Score a student attempt using a reference solution, then let it shorten its own response with correctness checked only afterward.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does SRCL rewriting differ from showing a teacher gold to rewrite?
Recursive Self-Improvement via On-Policy Distillation for Reasoning
Show answer and example
Answer: Rewrite sees only problem and its own response. Gold is for acceptance. Require shorter, self-contained, naturally ended and correct outputs to avoid mistaking privileged copying for autonomous revision.
Illustrative example: Do not include the reference solution in the shortening prompt.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
What if no shorter revision qualifies in Recursive OPSD?
Recursive Self-Improvement via On-Policy Distillation for Reasoning
Show answer and example
Answer: Run DCE only; don't force failed or longer rewrites into SFT. Identify prompt-role differences in frozen teacher controls; the local assistant-side arm isn't the original user-side OPSD baseline.
Illustrative example: A correct but longer answer does not pass SRCL and must stay excluded.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why add temporal credit to Gamma-OPD instead of only immediate log gaps?
Show answer and example
Answer: Choices shape future sequence. Discounted future teacher gaps inform current credit, with verifiable return constraining direction. This isn't only immediate imitation.
Illustrative example: An early branch can cause several later divergences that its local log gap underestimates.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why propagate teacher credit backward and add verifiable advantage?
Show answer and example
Answer: Later teacher evidence captures effects of early actions; verified outcomes align imitation with task returns. Teacher preference isn't correctness. Local formula checks don't establish capability.
Illustrative example: An early bad choice may cause several later gaps; credit it upstream while still requiring final verification.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does discounted teacher credit differ from adding unbounded outcome reward?
Show answer and example
Answer: Discounting carries sequential distillation; bounded advantage controls strength of verified outcomes. Arbitrary reward weights change the balance without proving one better.
Illustrative example: A giant terminal reward should not overwhelm temporal guidance without limits.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
What does the local Gamma-OPD kernel verify?
Show answer and example
Answer: Discounted propagation and bounded reward, not full language model training. Preserve order, terminal masks and frozen teacher signals. Shuffling tokens cannot verify temporal credit.
Illustrative example: Moving future credit to another response loses trajectory meaning even when sums are computable.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why distinguish source and destination tokens in RouteOPD?
Distillation as Probability Transport: Routed On-Policy Distillation
Show answer and example
Answer: Student-overestimated items have excess mass, underestimated items need mass. Explicit transport identifies where corrections come from and go instead of indiscriminate matching.
Illustrative example: Move mass from overpreferred wording to a teacher-supported key digit.
Sections in the Chinese source: 背景与主要改动, 核心机制. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why isn't transport equivalent to boosting every high-probability teacher token?
Distillation as Probability Transport: Routed On-Policy Distillation
Show answer and example
Answer: Select sources/destinations by student over/underestimation relative to teacher, not absolute teacher mass. This traces corrections and conserves probability. Interpretability doesn't prove teacher correctness; control support and evaluation.
Illustrative example: An already popular word in both models may need no boost; a teacher-supported overlooked word can receive mass.
Sections in the Chinese source: 背景与主要改动, 核心机制. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does transporting mass differ from replacing the student distribution with teacher probabilities?
Distillation as Probability Transport: Routed On-Policy Distillation
Show answer and example
Answer: Transport addresses selected discrepancies, not every coordinate. Require nonnegativity, conservation and defined source/destination; ordinary softmax isn't routing.
Illustrative example: Unrouted candidates can remain unchanged, unlike full replacement.
Sections in the Chinese source: 背景与主要改动, 核心机制. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
What invariants should RouteOPD transport check?
Distillation as Probability Transport: Routed On-Policy Distillation
Show answer and example
Answer: Never remove more mass than available; additions equal removals; final probabilities nonnegative and normalized. No transport at zero gap. A kernel doesn't reproduce teacher-model training.
Illustrative example: Removing one unit and adding only half breaks normalization.
Sections in the Chinese source: 背景与主要改动, 核心机制. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why use a verifier to correct OPDVR teacher reward signs?
OPDVR: On-Policy Distillation with Verifiable Rewards
Show answer and example
Answer: Teacher/student log ratios can oppose trajectory correctness. Retain only sign-consistent credit to avoid reinforcing errors or suppressing correct trajectories.
Illustrative example: Do not punish a different correct solution just because the teacher dislikes it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why retain only nonnegative teacher signal for success and nonpositive for failure?
OPDVR: On-Policy Distillation with Verifiable Rewards
Show answer and example
Answer: Token ratios can conflict with terminal outcome. One-sided ReLU uses verifiable correctness to gate implicit reward, not add a standalone RL loss. Coarse labels don't prove each token and remain scoring-only.
Illustrative example: Suppress a negative imitation signal for a correct alternative without requiring teacher wording.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does OPDVR differ from adding another RL loss?
OPDVR: On-Policy Distillation with Verifiable Rewards
Show answer and example
Answer: It removes the inconsistent part of implicit reward rather than mixing two losses with compensating weights. The objectives differ.
Illustrative example: Zero out positive teacher reward on an error rather than add a negative term hoping it cancels.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Which half-axis does OPDVR retain for correct and wrong trajectories?
OPDVR: On-Policy Distillation with Verifiable Rewards
Show answer and example
Answer: Correct keeps nonnegative log ratios; wrong keeps nonpositive; zero stays zero. Verifier is a fixed training signal, not rollout input. Local candidates aren't Qwen math rollouts.
Illustrative example: A positive log ratio on a failure becomes zero, not a sign-flipped extra penalty.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Should OPDSearch+ students imitate search-result text as their own output?
OPDSearch+: Search-Enhanced On-Policy Distillation with Reinforcement Learning
Show answer and example
Answer: No. Students generate queries, reasoning and answers; search observations condition later actions. Teacher guidance doesn't make tool output a student target.
Illustrative example: Use a webpage excerpt to ask the next query rather than role-play the returned page verbatim.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why distill from a frozen general teacher before continuing agent RL?
OPDSearch+: Search-Enhanced On-Policy Distillation with Reinforcement Learning
Show answer and example
Answer: Dense guidance on student search traces improves the starting behavior. Verifiable RL then adapts to environmental results rather than remain bound to teacher preferences. No specialized teacher training required; bias and proxy limits remain.
Illustrative example: Learn to ask workable queries, then let retrieved evidence and final verification adjust habits.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does on-student search distillation differ from offline teacher demonstrations?
OPDSearch+: Search-Enhanced On-Policy Distillation with Reinforcement Learning
Show answer and example
Answer: The teacher sees states and prefixes the student actually reached, including mistakes; offline continuations come from teacher states. Different recovery coverage isn't a universal performance ranking.
Illustrative example: Online scoring sees a wrong first query that a correct teacher demonstration never visits.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Does local OPDSearch+ execute real query, reasoning and answer tokens?
OPDSearch+: Search-Enhanced On-Policy Distillation with Reinforcement Learning
Show answer and example
Answer: No. Candidate-level process axes proxy generation/search feedback without real index/model. The real mask covers only generated text, never returned documents.
Illustrative example: A process score doesn't mean a Wikipedia search ran.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why estimate reasoning progress independently in R²-OPD?
Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress
Show answer and example
Answer: Teacher preference needn't help solve the task. Compare independent progress and mask conflicting supervision rather than equate style with reasoning.
Illustrative example: A fluent explanation may score high with teacher while moving farther from the solution.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why mask conflicting teacher and progress rankings instead of trusting teacher?
Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress
Show answer and example
Answer: Style can receive teacher approval without useful progress. Independent ordering checks direction. Masking signals uncertainty, not proof a span is wrong, and the progress evaluator still needs validation.
Illustrative example: Don't reinforce a polished long explanation if it leaves the key constraint unsolved.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does rank filtering differ from summing rewards?
Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress
Show answer and example
Answer: Filtering requires agreement to retain teacher signal. Addition lets a large score offset a poor one. Agreement doesn't prove either judge correct.
Illustrative example: A huge teacher score cannot buy back a span rejected by progress disagreement.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
What identities must both R²-OPD rankings share?
Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress
Show answer and example
Answer: Score the same spans of the same trajectories and align by identity; define ties. Local candidates do not run a real progress judge.
Illustrative example: Match a span across rankings, not the third list position.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why keep a reference alongside SR-OPSD self-teacher?
SR-OPSD: Self-Referenced On-Policy Self-Distillation
Show answer and example
Answer: Self-teacher can drift or become overly strong. Reference anchors the geometrically interpolated target and Rényi sensitivity control instead of trusting self-teacher alone.
Illustrative example: Retain some prior support when teacher strongly favors a new mode.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why geometrically mix self-teacher and reference?
SR-OPSD: Self-Referenced On-Policy Self-Distillation
Show answer and example
Answer: Log-space combination treats tails and disagreement differently from arithmetic mixtures. Reference anchors, teacher guides and divergence regulates projection sensitivity. Equal endpoints do not make two interpolation methods equivalent.
Illustrative example: Distributions favoring different tokens can give very different intermediate geometric and arithmetic targets.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does geometric interpolation differ from arithmetic averaging?
SR-OPSD: Self-Referenced On-Policy Self-Distillation
Show answer and example
Answer: Combine weighted log probabilities then normalize; arithmetic directly mixes mass. Low support from one side affects them differently, and formulas are not interchangeable.
Illustrative example: A nearly zero mass on one side can depress a geometric target more than arithmetic average.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Why form SR-OPSD targets in log space and normalize explicitly?
SR-OPSD: Self-Referenced On-Policy Self-Distillation
Show answer and example
Answer: Power products underflow. Weighted log-space softmax yields a stable distribution on aligned teacher/reference vocabulary. No full model/Rényi experiments locally.
Illustrative example: Raw elementwise product without normalization isn't a probability distribution.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why is U-OPSD privileged view still externally unsupervised?
Show answer and example
Answer: It comes from internal sampling consensus, without external answers/reward or a larger teacher. One view sees an extra internally produced signal, not independently verified truth.
Illustrative example: A short self-selected consensus solution can still be a shared error.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
How avoid treating shortest consensus as truth without gold?
Show answer and example
Answer: Consensus selects a reference and repair targets as pseudo-supervision, not correctness verification. Shared errors may be reinforced. Report internal agreement separately from external holdout and keep gold out of student inputs.
Illustrative example: Several responses can use the same wrong formula; the shortest still isn't a standard solution.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
Why distinguish shortest consensus answer from arbitrary shortest answer?
Show answer and example
Answer: First require majority agreement, then minimize length within eligible outputs. The shortest unrestricted may be an empty or outlier answer. Consensus isn't gold.
Illustrative example: A short erroneous refusal doesn't become privileged guidance because it saves tokens.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Can U-OPSD use gold before choosing an internal reference?
Show answer and example
Answer: No. Use predictions only for consensus, thresholds and length. Gold is for independent evaluation. Deterministic mini-suite isn't full multi-turn self-distillation.
Illustrative example: Filtering correctness before voting smuggles answer information into training.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why distill cross-lingually around reasoning pivots in RP-OPSD?
Show answer and example
Answer: Surface wording differs across languages; state-changing tokens matter more. Compare matched teacher distributions with/without the English reference to locate large information-induced shifts.
Illustrative example: A connective and a key equation need not receive equal credit based on text differences.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why compare matched views with and without English solutions?
Show answer and example
Answer: Reference-induced distribution shifts can identify reasoning transitions rather than superficial translation. Concentrate privileged distillation there and keep reference anchoring. Match all other inputs and keep solutions teacher-only.
Illustrative example: Translated connectives and words determining operation order may shift differently after adding a solution.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does pivot detection differ from uniformly imitating all English tokens?
Show answer and example
Answer: Uniform imitation can copy style and translation artifacts. Conditional shifts highlight important positions while retaining reference anchoring. Shifts aren't causal truth, so validate teacher quality.
Illustrative example: Different English wording with unchanged mathematical state need not rewrite every student word.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
What must the two RP-OPSD teacher views share?
Show answer and example
Answer: Same model, question and student prefix; remove only reference conditioning. Student rollout cannot see reference answers. Local deterministic proxies aren't real multilingual teachers/corpora.
Illustrative example: Changing both model and question prevents attributing probability changes to the English reference.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why distill the teacher's post-training change from base in OPD²?
On-Policy Delta Distillation for Multilingual Math Reasoning
Show answer and example
Answer: Teacher's preexisting preferences may not come from target training. The delta emphasizes what changed instead of transferring all language/style bias; matching checkpoints is essential.
Illustrative example: An unchanged long-sentence preference should not count as newly acquired skill.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why copy the delta rather than every teacher preference?
On-Policy Delta Distillation for Multilingual Math Reasoning
Show answer and example
Answer: Comparing before/after logp isolates directions associated with post-training instead of inherited language/style/knowledge. It needs comparable checkpoints and isn't proven pure capability. Local small distributions don't verify language transfer.
Illustrative example: If RL improves conclusions but politeness is unchanged, the delta avoids treating the greeting as a new skill.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does delta distillation differ from copying the post-trained distribution?
On-Policy Delta Distillation for Multilingual Math Reasoning
Show answer and example
Answer: It subtracts the teacher base; direct matching includes old knowledge and preferences. Subtraction may discard useful information or retain training noise, so not universally better.
Illustrative example: Learning the update is not copying the whole teacher knowledge base.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
May OPD² use arbitrary teacher families for its two probabilities?
On-Policy Delta Distillation for Multilingual Math Reasoning
Show answer and example
Answer: Not silently. Use paired base and post-trained checkpoint on the same prefix with aligned vocabulary/conditioning. Local candidates do not load multilingual math checkpoints.
Illustrative example: An unrelated architecture as base confounds family calibration with the training delta.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why propagate later distillation differences to earlier positions in DASH?
Show answer and example
Answer: Early choices have multi-step consequences. Local disagreement gates a backward recurrence with adaptive horizon, without extra teacher passes.
Illustrative example: An early fork causing repeated later errors should see some of their credit.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why control DASH horizon by local disagreement rather than propagate everything?
Show answer and example
Answer: Later teacher feedback may help or become unreliable after divergence. Detached gating varies effective horizon without extra teacher calls. Fixed long or short windows can mix irrelevant signals or miss important outcomes. Gate is not truth.
Illustrative example: Carry future credit across consistent steps but weaken propagation at a severe divergence.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does DASH's continuous gate differ from fixed discount?
Show answer and example
Answer: Fixed discount propagates equally by position; gate uses relative local disagreement to vary horizon. Proxy credit is not always correct and doesn't add teacher rollouts.
Illustrative example: Consequences at equal distance can propagate differently under different local disagreement.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
How should DASH handle boundaries and gate gradients?
Show answer and example
Answer: Recur backward with no next term at sequence end, detach gate and block padding or cross-response propagation. Candidate positions aren't long-token traces.
Illustrative example: Do not carry future credit from the next answer into the last token of this one.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why prioritize the first error in CausalOPD?
CausalOPD: First-Wrong-Step Supervision for Distilling Causal Chain Reasoning
Show answer and example
Answer: Later errors can follow from an early break. Short repair at its source avoids redundant punishment, then a curriculum moves evidence→mechanism→conclusion. Teacher localization must be reliable.
Illustrative example: An incorrect premise can make later coherent derivation wrong.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why locate the earliest wrong step and use short-horizon repair?
CausalOPD: First-Wrong-Step Supervision for Distilling Causal Chain Reasoning
Show answer and example
Answer: Later mistakes can inherit an earlier false premise. Correct the turning point rather than imitate final conclusions, then organize evidence/mechanism/conclusion. The teacher can misidentify it; proxy diagnostics aren't causal reasoning capability.
Illustrative example: Repair the first confusion of correlation with causation, not just the last sentence.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
What tradeoff separates short first-error repairs and whole-trajectory imitation?
CausalOPD: First-Wrong-Step Supervision for Distilling Causal Chain Reasoning
Show answer and example
Answer: Short windows focus local breaks and reduce irrelevant supervision; full imitation covers more but may diverge from student behavior. Incorrect localization can miss the real error.
Illustrative example: Fixing one mechanism link doesn't guarantee every conclusion is correct.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
May first-error search default to index zero if nothing is found?
CausalOPD: First-Wrong-Step Supervision for Distilling Causal Chain Reasoning
Show answer and example
Answer: No. Explicitly represent no error or unreliable judgment. Align step indices and keep privileged diagnosis out of inference. Local candidates don't implement the full causal teacher.
Illustrative example: An empty list must not invent a first-step error.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why use persistent signals rather than one token gap in PCSD?
Show answer and example
Answer: Local gaps are noisy; one whole-response weight is coarse. Accumulate support in adaptive windows, decay declining trends and continuously gate supervision.
Illustrative example: One high teacher score should differ from several supported consecutive steps.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why accumulate sustained teacher-favoring evidence rather than switch on single gaps?
Show answer and example
Answer: Single differences are noisy while whole-response weights hide position. Windowed exponential accumulation and decline decay combine smooth OPSD/GRPO gates. Persistence stabilizes signals without proving teacher correctness.
Illustrative example: An isolated overestimated word need not receive the same supervision as a consistently supported passage.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does PCSD continuous gating differ from a hard threshold?
Show answer and example
Answer: Sigmoid varies strength smoothly; accumulated trends allow the same present gap to receive different weights. It still relies on teacher quality, not a new verifier.
Illustrative example: Equal current gaps can differ if one has risen consistently and the other fallen.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
May PCSD accumulation continue across independent responses?
Show answer and example
Answer: No. Reset per response and apply valid masks. Teacher/gate statistics are fixed supervision; deterministic checks aren't long-reasoning training.
Illustrative example: Previous problem's ending support cannot become the next one's prior.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why keep a context-free anchor in Flux-OPD?
Show answer and example
Answer: Students can absorb fixed context, but replacing it can abruptly change targets. The anchor stays fixed and accepts relative corrections from evolving contexts rather than wholesale replacement.
Illustrative example: A new retrieval background can change one judgment without moving all language targets.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why use a fixed anchor and conflict-weighted context corrections?
Show answer and example
Answer: Replacing context can make targets jump. Relative corrections preserve the context-free base; geometric overlap weakens disagreement. Its normalization constant isn't arbitrary pairwise opinion distance. Local axes do not perform real extraction.
Illustrative example: For opposing suggested reasoning paths, retain the shared anchor and attenuate corrections rather than follow every new prompt.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
Why weaken Flux-OPD corrections when geometric teacher views conflict?
Show answer and example
Answer: Geometric normalization reflects overlap. Large conflict reduces effective correction instead of averaging incompatible suggestions forcefully. Agreement also doesn't prove correctness.
Illustrative example: Two confident opposing answers should not receive twice the trust.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Do local Flux reward axes reproduce real evolving-context extraction?
Show answer and example
Answer: No. They proxy teacher changes without video, medical or language context models. Build log-space targets relative to one anchor; don't mix differences from unrelated baselines.
Illustrative example: Two hand-built logit tensors don't implement document extraction and updates.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why view β-OPSD as a case of regularized policy optimization?
Show answer and example
Answer: Reference and privileged teacher define a geometric target; token-logit distillation replaces expensive policy optimization. β has the derivation's specific role, not arbitrary mixture weight.
Illustrative example: Same guidance with different regularization yields different effective targets.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why combine geometric teacher/reference interpolation with return-to-go?
Show answer and example
Answer: Interpolation controls target direction/strength; return-to-go controls temporal credit. Vanilla is a specific setting, not proof arbitrary logit averages solve the same objective. Local positions aren't long sequences.
Illustrative example: How much to trust teacher and how far later errors influence early choices are separate decisions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How does return-to-go differ from a current-token log gap?
Show answer and example
Answer: It accumulates discounted future gaps, assigning later consequences to current decisions. Longer horizons can add variance and depend on termination; not automatically more accurate.
Illustrative example: A single early error can cause several later deviations invisible to immediate gap.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Does local β-OPSD execute long CoT with its linear-position proxy?
Show answer and example
Answer: No. Candidate actions and linear returns replace tokens/trajectory positions. Freeze teacher/reference and preserve ordering; proxy results aren't Qwen LoRA evidence.
Illustrative example: Suffix sums over a few numbers don't demonstrate equivalent credit in real generation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why let Relay-OPD teacher briefly take over then return control?
Show answer and example
Answer: An invalid prefix can trap future learning. Short, budgeted intervention returns to a learnable region while student still acts after handoff.
Illustrative example: Teacher repairs a few critical steps, then student finishes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why not count a relayed trace as purely student on-policy?
Show answer and example
Answer: Teacher generates intervened actions and changes later student conditioning. It is a mixture. Budget keeps help local and provenance tracks writers. Include teacher cost and don't claim independent success.
Illustrative example: A student finishing after two teacher steps need not succeed unaided.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
How do relayed traces differ from native student rollouts?
Show answer and example
Answer: Teacher intervention changes action sources and states; log takeover interval and budget. Teacher-assisted success is not autonomous student success.
Illustrative example: A teacher bypassing a hard step requires separate no-teacher evaluation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
How does the Relay state machine prevent budget overrun?
Show answer and example
Answer: Track cumulative handoff lengths, enforce total and record takeover/return/termination. Local teacher is cached candidate distribution, not real Qwen or eight-GPU training.
Illustrative example: Once budget is spent, do not start a second handoff.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why distill an AR teacher on a diffusion student's own decoding path in OPDLM?
Show answer and example
Answer: Moving to bidirectional denoising can forget learned behavior. Random masks can miss confidence-decoding states. Actual student rollouts reduce mismatch with inference.
Illustrative example: Train on the reveal order actually visited, not only random masking.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why distill along actual bidirectional decoding rather than random masks?
Show answer and example
Answer: Confidence decoding visits different contexts and attention conversion can cause forgetting. Score student-visited states with frozen AR targets while retaining its causal condition. Different visible evidence cannot be equated positionwise arbitrarily.
Illustrative example: A confidence-first reveal produces contexts random-mask training might never encounter.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
Do AR teacher and bidirectional student see the same information at a position?
Show answer and example
Answer: No. AR teacher scores causal history, student denoises visible unmasked tokens. Align positions and state what each sees instead of treating conditioning differences as quality.
Illustrative example: Student may see a revealed token to the right while teacher sees only left context.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Where should OPDLM compute loss and does teacher update?
Show answer and example
Answer: Use specified masked/supervised positions, freeze AR teacher and align logits to student trajectories. Mini-suite does not fully convert/train a real bidirectional LM.
Illustrative example: Including already determined tokens changes denoising supervision.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why does GKD ask a teacher to score student-generated prefixes?
Show answer and example
Answer: Teacher paths can be idealized and miss visited errors. Sample student trajectories and ask for full teacher distributions at those prefixes, teaching recovery without only copying demonstrations.
Illustrative example: Teacher can guide next actions after the student's second step goes wrong.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Why does same-prefix teacher scoring reach student mistakes better than complete teacher answers?
Show answer and example
Answer: Fixed teacher trajectories avoid many student errors. Teacher distributions on sampled student prefixes reach those states; data fraction controls fixed/on-policy mixture. Coverage doesn't guarantee teacher recovery or determine divergence choice.
Illustrative example: A correct demonstration never visits the student's wrong step, while prefix scoring can help from that actual state.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
Is GKD student data fraction the same as IPD token mixing?
Show answer and example
Answer: No. GKD controls training-source mix of fixed and student traces. IPD mixes policies at each state. Dataset mixtures and one-step mixtures generally visit different trajectories.
Illustrative example: Half the batch from fixed answers is not choosing teacher with 50% probability at every token.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Why specify which distribution comes first in GKD KL?
Show answer and example
Answer: Forward/reverse KL and generalized JSD differ in coverage and concentration. Freeze teacher on identical student prefixes. Candidate teacher distributions aren't real T5 translation/summarization.
Illustrative example: KL(T||S) and KL(S||T) yield different gradients despite both being KL.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
Why favor reverse KL in MiniLLM?
Show answer and example
Answer: Small students may not cover all teacher modes. Reverse KL concentrates on student-supported reliable teacher regions, possibly missing modes. Sharpness isn't always desirable.
Illustrative example: Learn several reliable phrasings rather than spread over all teacher continuations.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
How does reverse KL affect small students' treatment of teacher tails?
Show answer and example
Answer: Student expectation focuses capacity on visited teacher-supported modes; forward KL covers teacher mass. Direction changes gradients and expectation. Preserve teacher-mixed stabilization; candidate checks aren't long-text distillation.
Illustrative example: A limited student may concentrate on a few reliable phrasings rather than cover every rare teacher expression.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
What coverage tradeoff separates MiniLLM from forward KL?
Show answer and example
Answer: Forward emphasizes teacher-mass coverage; reverse follows student-sampled modes. Teacher-mixed sampling and variance reduction also matter; direction alone isn't the full method.
Illustrative example: Few reliable modes versus many weak modes requires task evaluation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Whose distribution defines MiniLLM reverse-KL expectation?
Show answer and example
Answer: Student, with explicit teacher-mixed sampling and corrections in practice. Arbitrary fixed teacher text is not a substitute. Local candidates lack real tokenizer, long generation and GPT judging.
Illustrative example: Scoring teacher solutions with a KL-looking formula doesn't provide student on-policy coverage.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Basics
How does MPOPD mixing a rollout policy differ from mixing a student target?
Proposing Better Rollouts for On-Policy Distillation
Show answer and example
Answer: A learned proposal changes which prefixes are visited. The student still minimizes reverse KL to the teacher and corrects the old proposal/student gap. Training against a mixed target changes the loss without necessarily visiting teacher-supported paths.
Illustrative example: Example: Reweighting observed bad paths cannot generate a rare good path that was never collected. A changed proposal may sample it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Core mechanism
Which importance weight multiplies the student's KL at position j?
Proposing Better Rollouts for On-Policy Distillation
Show answer and example
Answer: Eq. 7 uses the one-step ratio w[j−1], with weight 1 at the first prefix. Use frozen old student and proposal probabilities of the sampled token and clip. Neither current-token ratio nor cumulative product is correct; no recomputation by the updating student.
Illustrative example: Example: Ratios 4, 0.25, 4 yield KL weights [1,4,0.25], not [4,0.25,4] or [1,4,1].
Sections in the Chinese source: 核心公式, 本地结果与负控. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Method comparison
Does fixed-eta geometric mixing equal a probability average or a fixed KL radius?
Proposing Better Rollouts for On-Policy Distillation
Show answer and example
Answer: Neither. It adds weighted log probabilities before normalization, not probabilities. Distributions vary by prefix, so the same eta produces different KL movement. A fixed local radius can imply prefix-specific eta; the global setting is a computational compromise.
Illustrative example: Example: Eta 0.4 barely moves a prefix where teacher and student agree but moves more mass when they conflict.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
LLM post-training / Trajectory distillation: students, proposals, and prefix distributions
Implementation details
Why don't real gradients establish full MPOPD integration?
Proposing Better Rollouts for On-Policy Distillation
Show answer and example
Answer: The tensor kernel has no sampler, LM copy or teacher call. Each round must reset proposal from the updated student and optimize it toward the new geometric target. Student KL and target are renormalized on the same teacher top-k support. Numeric checks do not verify trajectory origins, resets or extended training.
Illustrative example: Example: Cached old prefixes backpropagate but aren't current proposal rollouts. A small CUDA tensor check is not accuracy on 16 reasoning benchmarks.
Sections in the Chinese source: 数据协议与实现边界, 代码映射, 高成本完整实验. The example illustrates the mechanism.
Topic guide and reading order
Start with this question¶
If a student generates from its own mistaken prefixes at inference, why might imitating complete teacher answers miss those states?
Takeaway¶
Autoregressive students visit their own prefixes at inference. GKD generates with the student and asks the teacher to supervise distributions on those states, reducing train–test state mismatch. MiniLLM shows that divergence direction changes which teacher probability mass a student covers. Track where trajectories come from, which prefix both models see, and how discrepancies become updates. MPOPD learns a separate proposal that samples near the geometric mixture of teacher and student. Student loss, proposal loss, and frozen one-step ratios have different roles, so trajectory distillation need not sample directly from the current student.
Example¶
The teacher first computes area then a ratio; the student writes the wrong formula. SFT on teacher prefixes may never teach recovery. OPD can query teacher probabilities on the student's prefix, but the teacher might also follow the mistake. Better coverage does not guarantee correct supervision; check teacher quality, divergence, and validation.
Common pitfall¶
Scoring student trajectories is not generating a new teacher answer or transferring all teacher capabilities. KL values, finite gradients, and candidate diagnostics support local mechanisms. Lightweight GKD and MiniLLM do not reproduce full checkpoint training. Multiple numerical seeds are not a capability comparison.
Reading order¶
Read GKD to separate sampling from teacher scoring, MiniLLM for divergence direction and sampling stabilization, and Learning Signals to distinguish high loss from a useful learning direction.
- GKD · Original paper
- MiniLLM · Original paper
- Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals · Original paper
Check your understanding¶
Does training ten times on cached teacher answers make the data student-on-policy?
Show suggested answer
No. The trajectories remain fixed teacher samples. Check which student policy sampled, and whether the teacher supervised on student prefixes. Offline student replay still needs a distribution-shift and update contract; the label “student” does not make it current on-policy data.