Teachers and correction: when extra context can help¶
简体中文 · English
LLM post-training / 188 questions · 47 papers
Try answering each question before opening the explanation. The bank grows as papers and questions are added.
Read this topic's comparisons, analysis, and cross-paper Q&A
188 questions · 47 papers
LLM post-training / Teachers and correction: when extra context can help
Basics
Why ask whether the student can complete an AOPD teacher action?
When to Follow the Teacher: Advantage-Based On-Policy Distillation
Show answer and example
Answer: Teacher's favored next step may be beyond student continuation skill. Relative terminal advantage measures usable guidance from existing student rollouts without counterfactual generations.
Illustrative example: An advanced proof start is not useful if the student cannot continue it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why do both AOPD terms share a truncated teacher measure?
When to Follow the Teacher: Advantage-Based On-Policy Distillation
Show answer and example
Answer: The estimator combines terms that cancel to measure teacher-action advantage for student continuation. Clipping only one or renormalizing changes the objective. Use other same-prompt rollouts for LOO, with frozen teacher/behavior and no new counterfactual continuations.
Illustrative example: Evaluate a teacher-preferred opening against the student's actual returns rather than reinforce it solely from teacher likelihood.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does AOPD differ from multiplying KL by reward?
When to Follow the Teacher: Advantage-Based On-Policy Distillation
Show answer and example
Answer: It includes a sampled term and full-distribution term under the same unnormalized truncated teacher measure, plus independent leave-one-out returns. Simple reward multiplication loses that estimator structure.
Illustrative example: Weighting one sampled word doesn't reproduce subtraction of the vocabulary-wide term.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Can AOPD clip only the sampled term or replace rollout S with current student?
When to Follow the Teacher: Advantage-Based On-Policy Distillation
Show answer and example
Answer: No. Both terms share the unnormalized truncated teacher measure; denominator is frozen rollout S. Update only Sθ, with groups larger than one for independent LOO. Don't change behavior probabilities across steps.
Illustrative example: Re-softmax after truncation changes total mass and correspondence between terms.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why skip teacher correction for correct ME-OPD replies?
Edit, Don't Imitate: Minimal-Edit On-Policy Distillation for Multimodal Reasoning
Show answer and example
Answer: Successful traces retain task reward without stylistic interference. Failures alone select original tokens the teacher supports less. The teacher scores existing responses instead of writing new ones.
Illustrative example: Do not rewrite a valid alternative solution just because its style differs.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why skip correction for successes and select only some tokens from failures?
Edit, Don't Imitate: Minimal-Edit On-Policy Distillation for Multimodal Reasoning
Show answer and example
Answer: Preserve successful student behavior with task reward. Compare teacher and old student only for errors, correcting less teacher-supported tokens. Full group advantage precedes split. Reference solution and full reply are teacher training inputs only.
Illustrative example: A misread geometry label can receive focused suppression without reworking a successful alternative response.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How do ME-OPD and Projected Distillation produce corrections differently?
Edit, Don't Imitate: Minimal-Edit On-Policy Distillation for Multimodal Reasoning
Show answer and example
Answer: ME-OPD teacher-forces and suppresses chosen original error tokens. Projected Distillation generates a verified revision and compares edited spans via DPO. Scoring existing tokens and producing paired text differ.
Illustrative example: Only the latter needs a newly written answer aligned to the original.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Compute ME-OPD group advantages before or after splitting successes and failures?
Edit, Don't Imitate: Minimal-Edit On-Policy Distillation for Multimodal Reasoning
Show answer and example
Answer: Use all K rewards before splitting. A success-only group has constant reward. Use frozen old student for strict-greater mask and divide by selected correction tokens across the minibatch.
Illustrative example: Teacher and old student ties are not selected; do not remake mask from updated scores.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why align shared textual events rather than token IDs in CSD?
CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer
Show answer and example
Answer: Tokenizers differ in IDs, vocabulary and segmentation. Verify decoded character spans, then compare probability of common text events.
Illustrative example: One tokenizer can split a word in two; identical IDs wouldn't establish matching text.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why preserve REST across tokenizers rather than softmax common matches?
CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer
Show answer and example
Answer: Build common text events from verified decoded boundaries and chain probabilities. REST carries omitted mass; renormalization discards it and changes divergence. Don't align by position or IDs alone.
Illustrative example: Two tokens and one token may express one word, but unmatched continuation probability still exists.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
What does REST preserve that renormalizing common candidates removes?
CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer
Show answer and example
Answer: It retains probability of unmatched events instead of pretending their intersection is the whole distribution. Do not replace chain probability with a length mean.
Illustrative example: A small shared intersection may appear very similar after softmax while most mass was omitted.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Do CSD student gradients flow only through the final token of a chunk?
CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer
Show answer and example
Answer: No. Include prefix chain factors, final token and REST. Freeze teacher. Non-1:1 weights don't enter denominator count; fall back to CE for no KD/shared event and detach overall ratio.
Illustrative example: Ignoring the first of two pieces miscomputes its entire text event.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why compare evidence-preserving and degraded crops in VAG?
Reinforcement Learning with Visually Attributable On-Policy Self-Distillation
Show answer and example
Answer: Privileged corrections can mix visual evidence with style. Contrast matched visual conditions to estimate direction, retain agreeing corrections and bound modulation of verifier advantage.
Illustrative example: After cropping, check whether a teacher's revision follows changed visual evidence.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why accept only teacher corrections matching the visual contrast sign?
Reinforcement Learning with Visually Attributable On-Policy Self-Distillation
Show answer and example
Answer: Privileged crop changes may be stylistic. Require compatible evidence direction and use bounded modulation of verifier advantage. Only visual input changes in same EMA model; arbitrary text logits and local documentation do not reproduce the chain.
Illustrative example: If preference changes after privileged crop but not after occluding evidence, don't automatically treat it as visual guidance.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does VAG differ from fixed textual teacher logits?
Reinforcement Learning with Visually Attributable On-Policy Self-Distillation
Show answer and example
Answer: The definition needs the same EMA vision model under real paired views. Fixed logits only simulate the formula, not visual sensitivity or attribution.
Illustrative example: Two manually supplied scores aren't an occlusion experiment.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Can VAG's teacher branches be different models, and was this run?
Reinforcement Learning with Visually Attributable On-Policy Self-Distillation
Show answer and example
Answer: Not under the same protocol: one EMA teacher, only visual input changes. The local page describes equations without executing multimodal training. Keep strict sign agreement and verifier advantage.
Illustrative example: Comparing a large vision teacher to a small text model confounds architecture and evidence.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why interpolate APSD successes but extrapolate beyond teacher for failures?
Correctness-Dependent Asymmetric Target Distributions for Self-Distillation Policy Optimization
Show answer and example
Answer: Success should retain existing student preferences with gentle interpolation. Failures use stronger teacher-direction correction. Feedback selects a target regime, not guaranteed correctness.
Illustrative example: Use lighter changes after a correct response and stronger correction after failure.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why freeze only APSD teacher rather than detach the interpolated/extrapolated target?
Correctness-Dependent Asymmetric Target Distributions for Self-Distillation Policy Optimization
Show answer and example
Answer: Target depends on current student as well as teacher. Detach loses the student coupling that differs for success/failure. Forward equality doesn't imply backward equality. Feedback is privileged training context only.
Illustrative example: Identical target probabilities can yield different parameter updates when only one graph retains target dependence.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
Why isn't freezing APSD self-teacher the same as detaching its entire target?
Correctness-Dependent Asymmetric Target Distributions for Self-Distillation Policy Optimization
Show answer and example
Answer: Teacher q is frozen, but product-of-experts target also depends differentiably on p. Whole-target detach removes a defining coupling term, despite identical forward values.
Illustrative example: Detaching a target branch partly built from current student changes optimization, not merely memory.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Which PoE tensors retain APSD gradients, and how interpret default α?
Correctness-Dependent Asymmetric Target Distributions for Self-Distillation Policy Optimization
Show answer and example
Answer: Student and its target branch are differentiable, teacher detached. Use log space to avoid underflow. Correct α is between zero and one, error α above one. Local RLRF settings differ from paper RLVR.
Illustrative example: Local 0.5 and 1.5 aren't universal paper settings.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Is a SPOD decision one token or an action of multiple tokens?
Show answer and example
Answer: Protect a full state-conditioned action including native termination, not one token or an episode. Labels are episode-level but protection is decision-specific.
Illustrative example: Tool name, arguments and terminator form one action within a possibly multi-action success.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why protect full-action probabilities instead of token votes or masking the whole success?
Show answer and example
Answer: A specific action in a success is protected when teacher supports it less; probability combines all its tokens. Cancel distillation there and add a soft floor, leaving other actions learnable. Shared parameter changes can still erode protected actions.
Illustrative example: Teacher can like some tool-call tokens and dislike others, so token votes can oppose the whole-call probability.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does a SPOD soft floor differ from masking an entire successful episode?
Show answer and example
Answer: Only selected actions are protected; other decisions still distill. A soft penalty below threshold isn't a hard guarantee and shared updates can change protected behavior.
Illustrative example: One useful successful decision need not block all teacher learning in its episode.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Does SPOD compare temperature sampling probability or raw full-action probability?
Show answer and example
Answer: Use raw pre-temperature/pre-constraint probabilities including termination, sum token logp. Recompute gate pre-update but detach it; average by action, state and episode so long traces don't dominate.
Illustrative example: Mean token probability isn't full-action probability and changes protection by length.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why organize TreeOPSD shared prefixes as a tree?
Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
Show answer and example
Answer: Different continuations from one history reveal outcomes. Back up leaf rewards with behavior likelihood into local credit and count shared turns once.
Illustrative example: Shared first three steps don't get duplicated because later branches differ.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why convert backed-up tree credit into a bounded Bernoulli target?
Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
Show answer and example
Answer: Rewards set direction but no stopping distance. A fixed finite target stops or reverses gradients when reached or exceeded. Deduplicate shared turns and weight branches by full actor-span likelihood, not average token score.
Illustrative example: Repeatedly training one success group stops when it has learned enough instead of reinforcing indefinitely.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does bounded Bernoulli teaching differ from repeatedly applying fixed advantage?
Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
Show answer and example
Answer: Frozen posterior gives a finite target: gradient stops at the desired relative change and reverses beyond it. Fixed advantages can keep pushing under repeated updates.
Illustrative example: After sufficient adjustment, stop rather than drive the action farther on the same batch.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Is TreeOPSD's Bernoulli a full next-token teacher?
Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
Show answer and example
Answer: No. A two-point posterior describes a sampled turn relative to old policy. Use full actor-span likelihood and leaf-reward sample SD, exclude observations, and retain bounded-target derivation.
Illustrative example: The two outcomes are not two vocabulary tokens.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why train a simple Searcher–Answerer teacher first?
MultiAgent-OPD: End-to-End Training of Hierarchical Multi-Agent Systems with On-Policy Distillation
Show answer and example
Answer: Team terminal rewards are coarse for search. Train a task-relevant local teacher, then freeze it to guide hierarchical Searcher. Leader and Answerer keep distinct outcome signals.
Illustrative example: Final answer correctness alone can't rate the search summary reliably.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why train a simple teacher but restart the hierarchical student from base?
MultiAgent-OPD: End-to-End Training of Hierarchical Multi-Agent Systems with On-Policy Distillation
Show answer and example
Answer: Searcher–Answerer learns local task guidance. Restart distinguishes teacher knowledge from hierarchy learning. Shared weights still receive role-specific signals, and Searcher teacher sees real subtask/history, not global gold for all.
Illustrative example: A failed answer can reflect delegation or retrieval; uniform terminal credit misses local causes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
Why restart phase two from the same base instead of continuing phase one?
MultiAgent-OPD: End-to-End Training of Hierarchical Multi-Agent Systems with On-Policy Distillation
Show answer and example
Answer: Restart isolates teacher learning from hierarchical student improvement. Shared initialization doesn't imply both teacher and student keep updating.
Illustrative example: Build a search teacher first, then train a team from base rather than convert teacher weights directly.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Do shared MultiAgent parameters still need role-specific loss masks?
MultiAgent-OPD: End-to-End Training of Hierarchical Multi-Agent Systems with On-Policy Distillation
Show answer and example
Answer: Yes. Roles use distinct advantages and KL. Searcher log ratio isn't causal contribution; mask tool observations. Activity counts environment steps with search, not raw calls.
Illustrative example: Multiple parallel searches in one step should not inflate leader activity without limit.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why use a waypoint checkpoint from the same teacher RL run?
Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation
Show answer and example
Answer: Endpoints hide reversal. Midpoint shows early/late directional consistency and gates endpoint credit without becoming another imitation target.
Illustrative example: Monotonic movement toward B and movement toward A then B can end similarly.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why use the midpoint as gate rather than a separate target?
Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation
Show answer and example
Answer: Equal endpoint shifts can be consistent or reversed. Compare student-weight-centered early/late directions, then scale endpoint credit only. Freeze gate with low-norm fallback; no requirement to imitate the midpoint.
Illustrative example: Two teachers raise one token eventually, but one first lowers it. Midpoint distinguishes trajectories without teaching that interim state.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does positive cosine consistency differ from endpoint magnitude?
Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation
Show answer and example
Answer: Magnitude can hide oscillation. Cosine checks alignment of centered early/late shifts. It is evidence about one training path, not correctness or arbitrary cross-run chronology.
Illustrative example: Opposite segments lower gate despite a large final shift.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Why default the waypoint gate to one for tiny norms?
Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation
Show answer and example
Answer: Undefined direction is not conflict. Fall back to one, with student-top-k weighted centering and valid-prefix batch scaling. Freeze support, teacher and advantage. Unsampled candidates may receive gradients.
Illustrative example: Near-zero early displacement shouldn't delete all endpoint credit because cosine is unstable.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why might selecting teachers tokenwise help compared with one per response?
Compatibility Matters: A Framework for Token-Level Teacher Selection in On-Policy Distillation
Show answer and example
Answer: A response mixes expertise and teacher–student compatibility can vary by prefix. Routing is a plausible mechanism, not a universal empirical win.
Illustrative example: Code and math explanations can benefit from different teachers.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why aren't average teacher probabilities and average logits equivalent?
Compatibility Matters: A Framework for Token-Level Teacher Selection in On-Policy Distillation
Show answer and example
Answer: Arithmetic mixes allocate mass across distributions; log averaging and normalization favor shared support, particularly under tail/mode disagreement. Confidence, distance and aligned features are distinct criteria without universally optimal routing.
Illustrative example: Math/code experts can favor different continuations; sharing their mass differs from requiring common support.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
What assumptions underlie confidence, Hellinger and hidden-distance routing?
Compatibility Matters: A Framework for Token-Level Teacher Selection in On-Policy Distillation
Show answer and example
Answer: Confidence values certainty, distribution distance values compatibility, and hidden distance requires aligned representations. They measure different properties; unequal dimensions can't be subtracted, and closeness isn't correctness.
Illustrative example: A confident teacher far beyond student skill may receive different weights by rule.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Does soft teacher routing mix probabilities or logits, and what if τ=0?
Compatibility Matters: A Framework for Token-Level Teacher Selection in On-Policy Distillation
Show answer and example
Answer: Arithmetic mix probabilities; averaged logits give a geometric target. τ=0 uses hard argmax. Freeze routing/target. Local sampled-token path is not Top-K KL.
Illustrative example: Different teacher modes get different mass under probability and logit mixtures.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why does training-free TF-OPD still need teacher-generated data?
Training-Free On-Policy Distillation
Show answer and example
Answer: Closed-form linear merging uses input second moments from each teacher's visited states to determine output matching. No gradients does not mean no generation or calibration.
Illustrative example: Two specialists' statistics reveal different input subspaces.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why use teacher-specific input second moments rather than a shared covariance?
Training-Free On-Policy Distillation
Show answer and example
Answer: They weight output error along actually visited directions. Arbitrary shared statistics change least squares; termination filtering and token normalization matter. Gradient-free still costs generation, rescoring and solving.
Illustrative example: One shared covariance may ignore the domain-specific support of one expert.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does equal-Gram merging differ from summing task vectors?
Training-Free On-Policy Distillation
Show answer and example
Answer: Identical Gram statistics should yield a weighted average, not amplified sum. Orthogonal support can preserve separate functions. Behavior depends on statistics.
Illustrative example: Two identical experts should not produce twice the task update.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Whose responses estimate TF-OPD Grams?
Training-Free On-Policy Distillation
Show answer and example
Answer: Each teacher rescores its own responses after termination/loop filtering; normalize by tokens. Use FP64 linear solves, not explicit inverse, and clean hooks. Random covariance isn't teacher evidence.
Illustrative example: Scoring student-generated text changes teacher visitation statistics.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why not choose an SRD answer by majority vote?
Learning from Unverified Peers via Shared Residual Distillation
Show answer and example
Answer: Unverified peers can all be wrong yet contain useful local signals. Compare same-question peers with cross-question controls; retain minimum common signed residual instead of declaring a whole answer correct.
Illustrative example: Failed traces can still share one reasonable intermediate step.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why retain minimum same-sign peer residual without selecting one whole answer?
Learning from Unverified Peers via Shared Residual Distillation
Show answer and example
Answer: Compare same-question changes to control to isolate local task effects. Conservative common magnitudes limit individual overreach. Student sees only own prefix; agreement is not truth and target cannot reinforce itself as a peer.
Illustrative example: Keep a weakly supported common middle word without declaring an unverified conclusion correct.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does minmod residual differ from averaging peer logits?
Learning from Unverified Peers via Shared Residual Distillation
Show answer and example
Answer: Keep the minimum magnitude only when all signs agree; otherwise zero. Mean can let strong opinions swamp conflict. Shared agreement can still be biased.
Illustrative example: Two positive peers and one negative don't get a positive residual by simple majority.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Why can an SRD coordinate-capped surrogate be negative?
Learning from Unverified Peers via Shared Residual Distillation
Show answer and example
Answer: It caps signed coordinate contributions, not total KL. Use q0 centering weights, exclude target from its peers and freeze teachers/targets. Clamping negative coordinates changes the objective.
Illustrative example: A negative local coordinate may be valid even when a KL interpretation concerns the full distribution.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why sample reminder-OPRD students without reminders?
OPRD: On-Policy Reminder Distillation for Long-Horizon Coding Agents
Show answer and example
Answer: Deployment is unhinted. After failure a privileged teacher rescores actual prefixes using reminders to internalize constraints. Successful traces use self-imitation rather than teacher rewrites.
Illustrative example: A missed testing constraint can guide training scoring while inference sees only original task.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why expire OPRD local reminders after the relevant next turn?
OPRD: On-Policy Reminder Distillation for Long-Horizon Coding Agents
Show answer and example
Answer: A specific observation triggers a short-lived constraint. Global/local scopes differ; teacher rescores actual prefixes and deployment receives no reminder. Normalize success imitation and failure distillation separately.
Illustrative example: A dependency reminder from compilation should expire after the dependency is fixed, not remain pending forever.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How do global and post-observation local reminder scopes differ?
OPRD: On-Policy Reminder Distillation for Long-Horizon Coding Agents
Show answer and example
Answer: Global reminders persist; local ones apply only to the next turn after that observation and then expire. Permanent local reminders change unrelated conditions.
Illustrative example: A fix for one error belongs to the next attempt, not every later turn.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
How choose reminder-OPRD Top-100 and refresh teacher?
OPRD: On-Policy Reminder Distillation for Long-Horizon Coding Agents
Show answer and example
Answer: Rank shared support by elementwise max of both logp vectors, then normalize each on it. Refresh teacher at specified milestones, otherwise freeze. Average each trace then success/failure separately.
Illustrative example: Independent supports compare different events in coordinatewise KL.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why maintain a quality–diversity archive per EliteReplay problem?
EliteReplay: Preserving Diverse Solutions in RL Post-Training via Quality-Diversity Search
Show answer and example
Answer: Accuracy alone can shrink solution coverage. Keep the shortest verified solution per approach/structure cell to preserve distinct successful strategies rather than store ordinary IID replay.
Illustrative example: Separate algebra and geometry cells so the common strategy doesn't overwrite the other.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why index EliteReplay by realized approach and balance problem before cell?
EliteReplay: Preserving Diverse Solutions in RL Post-Training via Quality-Diversity Search
Show answer and example
Answer: Requested methods can differ from generated ones. Use actual descriptor and retain shortest verified answer in each cell. Problem-first balancing prevents archive size from allocating weight. This isn't IID success caching.
Illustrative example: A requested geometry solution using algebra belongs in algebra, or identical methods masquerade as diversity.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does shortest-per-cell MAP-Elites differ from a global shortest answer?
EliteReplay: Preserving Diverse Solutions in RL Post-Training via Quality-Diversity Search
Show answer and example
Answer: Cell-local replacement preserves other strategies. Store actual descriptors, not requested approaches, which may not match generated behavior.
Illustrative example: A proof requested as induction but generated by enumeration belongs to the enumeration cell.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Why balance problems before occupied cells?
EliteReplay: Preserving Diverse Solutions in RL Post-Training via Quality-Diversity Search
Show answer and example
Answer: Otherwise larger archives dominate replay. Average response tokens, mix archive and GRPO as specified, clear padding before exponentiating and use no test gold for archiving.
Illustrative example: More occupied cells should not give one problem ten times the sampling weight.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why mask a valid premise in CCE's teacher contrast?
Condition-Contrastive Extrapolation for On-Policy Distillation of Language Models
Show answer and example
Answer: A teacher may follow a mistaken student prefix instead of using conditions. Matched ablation reveals condition-sensitive directions for an extrapolated auxiliary target. Student always sees the full problem.
Illustrative example: Removing a numerical premise can identify which next tokens depend on it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why match student prefixes across views and keep full student normalization?
Condition-Contrastive Extrapolation for On-Policy Distillation of Language Models
Show answer and example
Answer: Fixed prefixes isolate premise removal. Teacher auxiliary target is normalized on candidate support but student uses full-vocabulary softmax, retaining outside gradients. Gate controls only auxiliary correction; base OPD stays.
Illustrative example: If premise and student reasoning both change, likelihood differences can't isolate the premise.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
Does disabling CCE auxiliary gating remove all OPD at a position?
Condition-Contrastive Extrapolation for On-Policy Distillation of Language Models
Show answer and example
Answer: No. Sampled PPO-OPD remains; only auxiliary forward KL is removed. Premise eligibility and entropy gate the extra term.
Illustrative example: A low-entropy format token still receives base distillation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Where is CCE entropy computed, and is student renormalized on support?
Condition-Contrastive Extrapolation for On-Policy Distillation of Language Models
Show answer and example
Answer: Compute entropy on full teacher vocabulary before top-set union. Normalize auxiliary target within support but preserve student full softmax for outside-support gradients. Denominator counts all valid positions.
Illustrative example: Student support renormalization would hide outside mass and change loss.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why require note-induced replays to improve RSHD pass rate?
Failure Is the Mother of Success: Recovering Supervision with Grounding Hindsight
Show answer and example
Answer: Plausible reflection may not improve behavior. Independently resample and accept only strict improvement over original group, then distill new continuations rather than old failures.
Illustrative example: A note suggesting unit checks is rejected if retries still all fail.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why generate fresh replays and require improvement rather than distill old failures?
Failure Is the Mother of Success: Recovering Supervision with Grounding Hindsight
Show answer and example
Answer: Verify the note with actual new behavior and a strict pass-rate gain. Uniformly sample accepted replays and compare note/no-note positive lift on identical prefixes. Old failures aren't fixed by explanation; no gold in notes and no auxiliary term without acceptance.
Illustrative example: A convincing postmortem with all-failed new attempts is not valid teacher guidance.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does uniform replay selection differ from keeping successes only?
Failure Is the Mother of Success: Recovering Supervision with Grounding Hindsight
Show answer and example
Answer: Uniform selection preserves the note-induced distribution. Success-only conditions on outcomes a second time after note-level gating and changes provenance.
Illustrative example: An accepted replay group can include failures; don't silently retain only the correct answer.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Do zero-lift RSHD tokens count in the auxiliary denominator?
Failure Is the Mother of Success: Recovering Supervision with Grounding Hindsight
Show answer and example
Answer: Yes. Count all valid accepted replay tokens, including zeros. No accepted note gives exactly zero auxiliary loss. Detach teacher/lift and never construct notes from gold.
Illustrative example: Dividing only by positive-lift count would overamplify sparse weights.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why create LoOPSD teachers by repeating layers rather than using bigger weights?
LoOPSD: On-Policy Self-Distillation with Looped Teachers
Show answer and example
Answer: Shared parameters can compute a better distribution through a longer loop. Search contiguous repeat regions and distill on student prefixes. Extra compute, not external answers, supplies guidance.
Illustrative example: Repeating middle layers changes per-token computation without adding parameters.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why does repeated-layer LoOPSD cost compute and need validation route selection?
LoOPSD: On-Policy Self-Distillation with Looped Teachers
Show answer and example
Answer: Loops change distribution without new weights but execute more layers per token and search multiple routes. Select validation improvements over standard route, then distill student prefixes; never choose on test. More compute doesn't guarantee gains.
Illustrative example: Rereading a chapter may help but takes time; choose using practice tasks, not final exam answers.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does re-searching loops each round differ from a fixed initial loop?
LoOPSD: On-Policy Self-Distillation with Looped Teachers
Show answer and example
Answer: Student updates can change useful repeat regions. Search adapts, and continue only if independent validation beats standard routing, not test-selected routes.
Illustrative example: A useful repeated middle region can stop helping after the student learns it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Do teacher weights/routes get gradients during LoOPSD distillation?
LoOPSD: On-Policy Self-Distillation with Looped Teachers
Show answer and example
Answer: Freeze both for the round; update standard student via RKL. Training references can be used only for permitted route search, not student rollout. Count repeated-layer compute.
Illustrative example: Five repeated layers are not free even without new parameters.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why query unrouted teachers in Road-MOPD?
Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation
Show answer and example
Answer: Shared parameters can degrade other domains even if examples route separately. Evaluate other teachers on the same prefixes for compatibility and real auxiliary gradients.
Illustrative example: Math updates can affect code behavior despite separate data routing.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why use unrouted teachers and impose a target gradient norm floor?
Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation
Show answer and example
Answer: Auxiliary teachers check cross-domain harm on identical states. Gate token credit, then address real gradient conflicts without overweakening target learning. Norm floors are tradeoffs, not universal gains; arbitrary vectors don't replace objectives.
Illustrative example: Code guidance can protect prefixes during math updates, but full removal might stall math learning.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
Can Road token gating replace gradient projection?
Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation
Show answer and example
Answer: No. Gating scales local supervision; projection handles shared-parameter conflict with a target norm floor. A weight formula isn't the complete mechanism.
Illustrative example: Compatible scores do not rule out gradient conflict.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Why not remove the entire conflicting component in Road projection?
Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation
Show answer and example
Answer: Choose maximum feasible projection while preserving the target norm floor; full projection can violate it. Compute actual auxiliary gradients, not static scores or registry labels.
Illustrative example: Near antiparallel gradients can make a fully projected update vanish.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why align decoded spans for cross-tokenizer CAST?
CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation
Show answer and example
Answer: Token IDs and boundaries differ. Match equivalent decoded spans, then project teacher span logp into student positions.
Illustrative example: Two Chinese tokenizations may split a word differently but decode to the same span.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why align decoded spans before projecting teacher log probabilities?
CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation
Show answer and example
Answer: Position-wise IDs lack a common event. Verified text spans provide corresponding regions, then uncertainty/smoothness allocate conserved credit. Anchor unmatched text and disclose alignment gaps.
Illustrative example: If one teacher token becomes several student tokens, verify identical decoded span before allocating its score.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does CAST allocation differ from CSD shared events?
CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation
Show answer and example
Answer: CAST distributes total teacher span credit over student positions. CSD builds common-event probability distributions with REST. They conserve different objects.
Illustrative example: Conserving one span's advantage is not preserving all event probability mass.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
How does CAST uncertainty weighting preserve chunk credit?
CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation
Show answer and example
Answer: Normalize weights from uncertainty and local smoothness, then multiply chunk total advantage. Anchor unmatched positions to initial student rather than drop them and claim full coverage.
Illustrative example: A higher token share must not increase total chunk advantage.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why measure sustained teaching activity rather than similarity in SGUID?
Which Skill to Distill? SGUID: Selecting a Compact Skill Bank for Model–Skill Co-Evolution
Show answer and example
Answer: Relevant skills can teach poorly. Generate from verified traces and check early/late persistence and verifier agreement. Retrieval frequency isn't useful teaching count.
Illustrative example: A frequently retrieved but misleading skill shouldn't stay because it looks relevant.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why restore round-start weights and retrain only selected skills?
Which Skill to Distill? SGUID: Selecting a Compact Skill Bank for Model–Skill Co-Evolution
Show answer and example
Answer: Otherwise broad probe exposure confounds the compact curriculum effect. Reset, train selected skills, and regenerate from the updated model next round. Count sustained verifier-aligned teaching, not semantic matches.
Illustrative example: Testing ten hints gives extra practice; without a reset, later gain can't be assigned to the two selected hints.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
Why restore initial round weights after SGUID probing?
Which Skill to Distill? SGUID: Selecting a Compact Skill Bank for Model–Skill Co-Evolution
Show answer and example
Answer: Probes already update the model. Restore to isolate training with compact selected library from extra probe learning. Next round generates new skills, not a fixed manual archive.
Illustrative example: Continuing after probing all candidates would retain discarded skills' effects.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
May SGUID delete negative coefficients, and how should negative results be handled?
Which Skill to Distill? SGUID: Selecting a Compact Skill Bank for Model–Skill Co-Evolution
Show answer and example
Answer: No. The objective allows negative teaching signals. Check early/late activity and quality. Validation decline after two executed rounds remains a negative result, not success because the loop ran.
Illustrative example: Early gains don't justify retention if late signals fall below thresholds.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why learn MetaOPD weights through a virtual student step?
MetaOPD: Meta-Learned Token Weighting for On-Policy Distillation
Show answer and example
Answer: Teacher gap does not reveal which signals improve tasks. Evaluate virtual updates against training-reference loss, use meta-gradients to learn weighting, then update the student with new weights.
Illustrative example: A large gap on a token may harm the reference task and deserve lower weight.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why need a differentiable virtual update instead of weighting by current loss?
MetaOPD: Meta-Learned Token Weighting for On-Policy Distillation
Show answer and example
Answer: High loss can be noise. A virtual step reveals downstream reference-loss effects and meta-gradients, then real student update uses learned weights. Isolate descriptors, optimizer state and training-only references.
Illustrative example: Compare whether emphasizing a stylistic high-loss word or a reasoning token better serves the reference task.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does differentiable virtual AdamW differ from scoring an actual step afterward?
MetaOPD: Meta-Learned Token Weighting for On-Policy Distillation
Show answer and example
Answer: Virtual step retains the graph to weight network for meta-gradients, while real update mutates state without that derivative. Optimizer state/order matter; arbitrary SGD isn't equivalent.
Illustrative example: Ask how weights caused the hypothetical change, not only whether a metric changed.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
What detaches in MetaOPD's 74-dimensional descriptors and weight transform?
MetaOPD: Meta-Learned Token Weighting for On-Policy Distillation
Show answer and example
Answer: Detach descriptors, not the weight network through the virtual step. Center per response, bound via tanh and normalize to mean one. Use new weights for real update; references come from allowed training data.
Illustrative example: Test references cannot supervise outer-loop weights.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why use synthetic scene coordinates as privileged Where-OPD input?
Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenes
Show answer and example
Answer: Generated scenes provide exact objects/positions for teacher while student sees image/question. Internalize spatial hints without coordinate access at deployment.
Illustrative example: Teacher knows positions; student learns left/right from pixels.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why give teacher textual coordinates while student receives only image and question?
Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenes
Show answer and example
Answer: Exact synthetic object/location metadata gives privileged scoring on student prefixes. Deployment lacks that metadata, so keep student inputs clean. Preserve spatial mask; random-logit tests don't prove spatial ability.
Illustrative example: Teacher can correct left-of-box relationships, but test students cannot be told coordinates.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does spatial privileged text differ from ordinary text distillation?
Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenes
Show answer and example
Answer: The method needs matching actual scene images and spatial annotations. Random text distributions don't verify visual integration; privilege doesn't prove student reliance on images.
Illustrative example: A correct description paired with the wrong image still yields a computable loss but meaningless guidance.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Does local Where-OPD generate scenes and load an MLLM?
Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenes
Show answer and example
Answer: No. It checks teacher KL under a spatial mask aligned to generated student positions, with frozen teacher. Random logits aren't visual evidence.
Illustrative example: Two hand-built left/right logits don't constitute image reasoning.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why require every prefix to satisfy average cost in Weakest Link?
The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learning
Show answer and example
Answer: A final mean can hide bad early reasoning diluted by easy later tokens. First violation is absorbing. Costs are teacher negative logp.
Illustrative example: Appending common words cannot undo an unsupported prefix.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why does an early violation persist instead of being diluted by easy later tokens?
The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learning
Show answer and example
Answer: Every prefix must meet teacher cost. Absorption blocks post hoc dilution; immediate counterfactuals vary only action at the real visited prefix. Teacher improbability still doesn't prove factual error.
Illustrative example: Adding common words after an unsupported derivation may lower final cost but cannot satisfy already violated prefixes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does absorbing penalty differ from a recoverable moving gate?
The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learning
Show answer and example
Answer: Violation persists even if later average improves. A rolling gate might reopen. This is a prefix constraint, not proof teacher cost equals task value.
Illustrative example: One high-probability next token doesn't clear the previous violation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
May counterfactual current actions use different historical costs?
The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learning
Show answer and example
Answer: No. Start from actual prefix cost and vary only current action. Exact full-vocabulary immediate gradients coexist with sampled future returns. No full LM trainer.
Illustrative example: Inventing a worse history for an unfavored action invalidates its comparison.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why distill representations in PR-OPD rather than only final probabilities?
PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning
Show answer and example
Answer: Hindsight skills create privileged hidden states. Align projected states at matching layers and response positions, then combine with task reward. Teacher skill is removed at inference.
Illustrative example: A retrospective tool principle can guide internal features without being deployment input.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why forward the same response with/without hindsight skill rather than imitate skill text?
PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning
Show answer and example
Answer: Compare how skill changes same-token/layer representations and transfer to an unprivileged student. Copying skill or teacher replies changes targets. Detach teacher, combine task reward and remove privilege at inference.
Illustrative example: A teacher can rescore an earlier decision using a validation reminder that the deployed student never sees.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How do two forwards differ from concatenating skill to student context?
PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning
Show answer and example
Answer: Run ordinary student and privileged detached teacher separately. Giving student skill changes deployment assumptions and can teach dependency. Representation matching isn't answer supervision.
Illustrative example: A tested student should rely on learned parameters when no skill text is supplied.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
What masks apply to layerwise PR-OPD alignment?
PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning
Show answer and example
Answer: Align valid tokens of the same response at corresponding layers, excluding observations/padding. Skills come from completed training traces, not gold plans. Local kernel isn't agent training.
Illustrative example: Do not align extra teacher skill tokens to student response positions.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why intervene only at MAESTRO's high-disagreement prefixes?
From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation
Show answer and example
Answer: Teacher rollouts are expensive. Top-k coverage, similarity and early-position weighting allocate limited guidance. Disagreement doesn't automatically mean student error.
Illustrative example: An early fork can matter more than a punctuation discrepancy.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why use both top-k coverage and Bhattacharyya similarity for interventions?
From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation
Show answer and example
Answer: One checks support inclusion, the other shared probability allocation. Disagreement and position weight prioritize scarce teacher compute. It is a clue, not correctness; local PDS doesn't generate a takeover.
Illustrative example: Two models can share identical top-k words while assigning very different probabilities.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
Why combine coverage and Bhattacharyya similarity?
From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation
Show answer and example
Answer: Coverage checks whether student candidates include teacher support. Similarity compares mass within shared support. Either alone misses differences; high PDS isn't teacher correctness.
Illustrative example: Identical candidates can have almost opposite weights.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Does local MAESTRO generate takeover rollouts?
From Dissonance to Orchestration: Teacher Intervention in On-Policy Distillation
Show answer and example
Answer: No. It computes top-k coverage, normalized similarity and PDS. Real intervention needs generation, state handoff and cost accounting; no measured call savings.
Illustrative example: Hand-entered probabilities can trigger a score without any teacher rollout.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Can LEGO-OPD grounding likelihood be read as the final answer probability?
LEGO-OPD: Factorized Teacher Composition for Multimodal On-Policy Distillation
Show answer and example
Answer: No. It is the visual evidence factor, combined with language prior and normalized. Distinguish prior, likelihood and final posterior, not interpret one score as answer confidence.
Illustrative example: Vision supports red; language constrains valid next words. Combine and normalize for token probabilities.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why combine a language prior and visual likelihood instead of averaging teachers?
LEGO-OPD: Factorized Teacher Composition for Multimodal On-Policy Distillation
Show answer and example
Answer: Language supplies general grammar/reasoning while vision modifies evidence through a product. This can avoid copying visual model language bias if its likelihood is reliable. Local target checks do not prove vision.
Illustrative example: Language makes a cup description grammatical; visual evidence increases the color it shows.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does setting grounding α=0 differ from using only vision?
LEGO-OPD: Factorized Teacher Composition for Multimodal On-Policy Distillation
Show answer and example
Answer: A valid zero-weight visual contribution leaves language prior. Vision-only would remove language, a different target. α scales evidence, not votes between two complete models.
Illustrative example: Multiplying visual log contribution by zero leaves language scores, not pure vision.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Does local LEGO load both language and visual experts?
LEGO-OPD: Factorized Teacher Composition for Multimodal On-Policy Distillation
Show answer and example
Answer: No. It only combines factors. Align candidates, add in log space, then normalize. Visual likelihood isn't a full VLM posterior; random logits aren't visual capability.
Illustrative example: Multiplying two vectors doesn't make any model see an image.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why train AdviSD advisor and freeze executor?
AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation
Show answer and example
Answer: Improve advice to a fixed strong executor, isolating advisor contribution. Reflection generates candidate advice; only advice affecting executor preferences enough enters self-distillation.
Illustrative example: Improve parameter-check reminders without changing the tool model weights.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why score a fixed executor reply in both advice conditions?
AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation
Show answer and example
Answer: Fixed reply controls generation randomness and isolates preference shift. Train advisor on threshold-passing reflections while executor stays frozen. A likelihood increase isn't task success and calibration depends on advice distribution.
Illustrative example: Rescore the same recorded tool call in both contexts rather than subtract independently generated answers.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
Is influential advice necessarily successful advice?
AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation
Show answer and example
Answer: No. Likelihood contrast measures preference shifts on one recorded response. Donor calibration screens weak effects, not whether the change improves outcomes; evaluate tasks separately.
Illustrative example: Advice can increase support for an existing wrong action.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
May AdviSD generate separate replies for with/without advice comparisons?
AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation
Show answer and example
Answer: Not in place of paired rescoring. Use the same response and mean valid-token logp; threshold from donor absolute contrast quantile. No local real executor.
Illustrative example: Different newly generated long and short answers confound advice effects.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why let OLIVE teacher continue from student-visited states?
Learning from Teacher Continuations at Student States
Show answer and example
Answer: Student mistakes differ from ideal teacher paths. Real prefixes supply deployment-like supervision for continuation/recovery instead of memorizing complete solutions.
Illustrative example: Continue from an unusual intermediate expression the student reached.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why don't OLIVE prefixes receive supervised loss?
Learning from Teacher Continuations at Student States
Show answer and example
Answer: The teacher continues the student's actual prefix, giving useful next-step examples without treating student mistakes as correct labels. Apply CE only to teacher tokens. Teacher quality and full-model experiments remain separate.
Illustrative example: Read a repairable wrong expression and learn only the corrected teacher continuation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does teacher continuation CE differ from scoring original student tokens?
Learning from Teacher Continuations at Student States
Show answer and example
Answer: Continuation adds newly generated teacher text as target; rescoring evaluates existing sampled actions/states. A scoring kernel doesn't produce new teacher continuations.
Illustrative example: Writing the next steps differs from rating the old steps.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Can OLIVE remove prefixes from input because they have no loss?
Learning from Teacher Continuations at Student States
Show answer and example
Answer: No. Prefix remains conditioning for continuation and student supervision; mask targets only. Handle concatenation, EOS and padding. Local tests verify masks without a closed teacher.
Illustrative example: Retain three mistaken steps as context without training to imitate them.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why add role-specific advantage in MAS-OPD?
MAS-OPD: On-Policy Distillation for Multi-Agent Systems
Show answer and example
Answer: Terminal team outcomes hide member responsibility. Target vs other-role teacher signals supply local collaboration credit. Verify attribution rather than punish everyone equally.
Illustrative example: Good retrieval and faulty summarization call for different credit.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why put conflict attribution in teacher view instead of changing every role's reward?
MAS-OPD: On-Policy Distillation for Multi-Agent Systems
Show answer and example
Answer: One team score doesn't identify corrections. Privileged teacher distinguishes target and other contributions. Extra attribution is training-only and local mask/advantage does not establish multi-model collaboration.
Illustrative example: A sound planner and an executor using the wrong tool shouldn't receive the same blame.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does teacher-only conflict information differ from giving full attribution to all roles?
MAS-OPD: On-Policy Distillation for Multi-Agent Systems
Show answer and example
Answer: Students act on their own public histories; only teacher sees extra conflict explanation. Giving hindsight to students changes deployment inputs and hides whether learning internalized it.
Illustrative example: Teacher may know who misunderstood after failure; deployed members cannot start with that diagnosis.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Does local MAS-OPD jointly train real models?
MAS-OPD: On-Policy Distillation for Multi-Agent Systems
Show answer and example
Answer: No. It tests role advantage and conflict masks. Align roles, traces and valid generated positions. Registry labels or misassigned signals don't establish team training.
Illustrative example: Swapping role order can keep arrays valid while assigning credit to the wrong member.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
What does language inertia mean in OPD-Aha?
OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation
Show answer and example
Answer: A wrong student prefix can lead a teacher to keep repeating it despite visual evidence. Compare actual and empty vision to recover visual contributions.
Illustrative example: If prior text says dog and picture shows cat, don't blindly continue dog.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why compare teacher distributions with real and empty visual input?
OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation
Show answer and example
Answer: The comparison isolates preference changes attributable to image conditioning despite mistaken text context. It helps separate evidence from prior but doesn't guarantee correctness. Local evidence is reconstruction only.
Illustrative example: A cat image can shift preferences away from the prefix's dog claim relative to no vision.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does real/empty vision contrast differ from raising confidence?
OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation
Show answer and example
Answer: It changes vision while matching teacher context. Sharpening can amplify language priors. Empty-view design and teacher quality affect attribution.
Illustrative example: If both views confidently say dog, sharpening won't discover the cat.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Does local OPD-Aha reconstruction use a multimodal checkpoint?
OPD-Aha: From Linguistic Momentum to Visual Reflection in Multimodal On-Policy Distillation
Show answer and example
Answer: No. It tests target math. Real implementation needs paired visual/empty inputs and identical student prefixes scored by a model. Fixed differences don't establish visual reflection.
Illustrative example: Manually increasing cat probability checks reconstruction, not image recognition.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why use self-generated mistakes as negative teaching material in NSD?
Negative Self-Distillation: Learning to Reason by Avoiding Flaws
Show answer and example
Answer: Failures identify directions to avoid. Negative teacher models error distribution and a dynamic gate concentrates suppression on key reasoning while protecting ordinary language.
Illustrative example: A misused proof rule can supply an avoid signal.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why not suppress the entire erroneous answer in NSD?
Negative Self-Distillation: Learning to Reason by Avoiding Flaws
Show answer and example
Answer: Negative teacher gives a distribution to avoid, with dynamic gates selecting reasoning-critical tokens while preserving useful language/steps. Selection doesn't prove errors and full LM training wasn't reproduced.
Illustrative example: A carry error needn't suppress every connective and correct digit.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does negative self-distillation differ from lowering every wrong answer token?
Negative Self-Distillation: Learning to Reason by Avoiding Flaws
Show answer and example
Answer: Wrong responses can contain valid facts, grammar and steps. Selective gating limits collateral damage, but isn't a verified step label.
Illustrative example: A wrong result doesn't mean all numbers and connectives should be forgotten.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
What supplies local NSD negative teachers and gates?
Negative Self-Distillation: Learning to Reason by Avoiding Flaws
Show answer and example
Answer: Candidate policies test control flow, not full-parameter LLM. Freeze and align gates to penalized positions; test gold cannot preidentify errors under no-leakage claims.
Illustrative example: Hand-labeled errors are a unit fixture, not autonomous error localization.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why does CompassOPD need each teacher's own reference?
CompassOPD: Cross-Family On-Policy Distillation via Within-Family Likelihood Shifts
Show answer and example
Answer: Absolute probability scales differ across families. Relative change captures post-training direction, and within-family centering reduces global bias before using the student anchor. It isn't correctness proof.
Illustrative example: A naturally sharp teacher shouldn't look more learned just because probabilities are larger.
Sections in the Chinese source: 背景与主要改动, 核心机制. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why compare post-trained teacher to its own reference rather than absolute teacher–student scores?
CompassOPD: Cross-Family On-Policy Distillation via Within-Family Likelihood Shifts
Show answer and example
Answer: Calibration and sharpness vary by family. Relative likelihood shifts and within-family centering extract direction, with a student reference anchor. Teacher bias can remain.
Illustrative example: Two teachers may both increasingly favor the right step despite very different initial distributions.
Sections in the Chinese source: 背景与主要改动, 核心机制. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does Compass trade off against averaging cross-family probabilities?
CompassOPD: Cross-Family On-Policy Distillation via Within-Family Likelihood Shifts
Show answer and example
Answer: Relative directions mitigate scale effects but discard some global information. Direct mixtures retain absolute posteriors. Compare under equal protocol; no universal win.
Illustrative example: Two moves toward the correct option shouldn't be obscured by native temperature differences.
Sections in the Chinese source: 背景与主要改动, 核心机制. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
What comparable sets should Compass center over?
CompassOPD: Cross-Family On-Policy Distillation via Within-Family Likelihood Shifts
Show answer and example
Answer: Use consistent candidates/weights for specified family changes and a separate student reference. Don't center arbitrarily across questions. Local audit magnitudes aren't training success.
Illustrative example: Mismatched option order makes subtraction meaningless.
Sections in the Chinese source: 背景与主要改动, 核心机制. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
What information transfers from weak to strong in OPRD?
Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Show answer and example
Answer: Weak post-training logit change relative to base augments strong verifier gradient. Weak answers needn't outperform strong ones; a useful update direction may transfer.
Illustrative example: A learned unit-check tendency can help without copying weak arithmetic.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why transfer weak-model direction without asking the strong model to imitate answers?
Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Show answer and example
Answer: Extract centered normalized weak-after/base difference and boost aligned strong-verifier components. The transfer object is direction, not content. Utility requires alignment and candidate diagnostics don't prove broad gains.
Illustrative example: A stronger model can learn to check units without copying an incorrect weak numeric result.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does OPRD gradient augmentation differ from full weak-teacher distillation?
Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Show answer and example
Answer: Keep strong verifier gradient and amplify its component along weak learning direction, instead of fitting weak distribution. An irrelevant direction may not help.
Illustrative example: Strong model still follows its reward with directional modulation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
How handle zero change and gradient direction in OPRD?
Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Show answer and example
Answer: Center and unit-normalize deltas; treat zero norm as no direction. Add projection to original gradient, not replace it. Local candidates only map the proxy to parameters.
Illustrative example: Identical weak checkpoints should not generate a divide-by-zero teaching direction.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why can weak-model prefixes improve strong exploration?
Boosting LLM Exploration via Weak-Model Guidance in RLVR
Show answer and example
Answer: They move the target to states its own policy rarely visits, expanding reasoning coverage. Prefixes can be wrong; independent verification still defines reward.
Illustrative example: Try continuing an alternative decomposition and check the result.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why mix weak prefixes even though they can be wrong?
Boosting LLM Exploration via Weak-Model Guidance in RLVR
Show answer and example
Answer: They provide uncommon exploration starts. Entropy truncation limits unstable continuations and native traces maintain coverage. Hints aren't labels; use independent verifier. Frozen proxy isn't dual-model training.
Illustrative example: An alternate decomposition can be tried but still receives failure if validation fails.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does prefix guidance differ from SFT on complete weak answers?
Boosting LLM Exploration via Weak-Model Guidance in RLVR
Show answer and example
Answer: It supplies only a starting point; target generates and learns from reward. Native/guided mixtures and entropy truncation control coverage instead of trusting weak solutions wholesale.
Illustrative example: Give an opening idea without asking the student to copy a wrong conclusion.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Is the local frozen-reference proxy actual dual-checkpoint RLVR?
Boosting LLM Exploration via Weak-Model Guidance in RLVR
Show answer and example
Answer: No. Candidate distributions simulate guidance. Record mixture proportions and prefix boundaries; keep hints out of independent test. No real two-model generation.
Illustrative example: Numeric alternative starts don't measure linguistic weak-prefix quality.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why does Video-OPSD teacher see evidence frames while student sees full video?
Show answer and example
Answer: Privileged frames focus teacher supervision, while student learns to find relevant evidence amid distractions and does not receive annotation at inference.
Illustrative example: Teacher can see the cup pickup moment; student must find it within the full activity.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why keep evidence frames teacher-only and full video for student?
Show answer and example
Answer: Human-labeled frames give focused supervision, weighted by token dependence. Student must handle distractions as deployed; supplying frame labels changes the task. No-video diagnostics aren't vision evidence.
Illustrative example: To answer who picked up the cup first, student still needs to locate the relevant segment.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How do evidence-dependent weights differ from equal token KL?
Show answer and example
Answer: Emphasize positions influenced by visual evidence, not all style tokens equally. Dependency isn't factual certainty. No-video smoke doesn't validate paper training time.
Illustrative example: An action word can depend more on frames than a connective.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Does local Video-OPSD smoke include video frames?
Show answer and example
Answer: No. Independent reward axes proxy evidence. Available checkpoints don't imply trained video model. Align teacher targets to student prefixes and keep annotations out of deployment.
Illustrative example: A synthetic process score doesn't show the model watched a video.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why expand temperature coverage before UC-MOPD selection?
Preserving General Capabilities during Domain Specialization with Uncertainty-Calibrated MOPD
Show answer and example
Answer: Ordinary sampling may yield few positive-credit positions. Dual temperatures broaden exploration, then trajectory density/token reliability filter it. More candidates aren't all correct.
Illustrative example: Try another solution first, then check whether its steps help.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
What do temperature expansion, trajectory selection and token gating each solve?
Preserving General Capabilities during Domain Specialization with Uncertainty-Calibrated MOPD
Show answer and example
Answer: Expand candidate coverage, favor trajectories with dense positive advantages, then gate individual updates with entropy-calibrated probabilities. More samples don't guarantee correct supervision; no full teachers locally.
Illustrative example: Sample varied solutions, select one with useful steps and reinforce only reliable tokens.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How do positive-advantage density and CLL gating work together?
Preserving General Capabilities during Domain Specialization with Uncertainty-Calibrated MOPD
Show answer and example
Answer: Density selects useful trajectories; CLL controls local tokens inside selected traces. Both are required and confidence isn't truth.
Illustrative example: A generally useful solution can still contain positions not worth reinforcing.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
What does stochastic UC-MOPD gating need to log, and were checkpoints loaded?
Preserving General Capabilities during Domain Specialization with Uncertainty-Calibrated MOPD
Show answer and example
Answer: Log probabilities, seed and valid positions so random draws aren't deterministic decisions. No general/domain teachers loaded; formula checks don't prove retained general capabilities.
Illustrative example: Track each differing random mask, not just report the best draw.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why are relevant time intervals privileged training information in Clue-OPSD?
Where to Look Matters: On-Policy Self-Distillation for Long-Video Understanding
Show answer and example
Answer: A focused teacher can guide evidence retrieval in long video. Student still receives full video/question and loses clue and teacher at deployment.
Illustrative example: Teacher knows the opening-door timestamp; deployed student must find it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why query a clue-conditioned teacher on the student's own prefixes?
Where to Look Matters: On-Policy Self-Distillation for Long-Video Understanding
Show answer and example
Answer: Actual student prefixes reduce trajectory mismatch while clue segments focus evidence. Clues are training privilege; at inference the student must handle full video without markers, answers or teacher.
Illustrative example: Guide correction of an incomplete event ordering during training, without giving the manual clip at deployment.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does a clue teacher differ from shortening the student's video?
Where to Look Matters: On-Policy Self-Distillation for Long-Video Understanding
Show answer and example
Answer: Shortening input simplifies the task; teacher-only privilege changes supervision while preserving full-video difficulty and costs.
Illustrative example: Seeing a solution clip and seeing all footage are not matched evaluations.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Why score Clue-OPSD KL on student rather than teacher prefixes?
Where to Look Matters: On-Policy Self-Distillation for Long-Video Understanding
Show answer and example
Answer: Guide actually visited states with aligned token positions despite different visual conditions. The mini-suite does not execute a full video-checkpoint experiment.
Illustrative example: If the student misidentifies a person, teacher must rate that prefix, not its own ideal opening.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why specialize then merge in SMOPD?
SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation
Show answer and example
Answer: Reward priorities produce complementary teachers. Merge dense supervision on student-visited states instead of forcing conflicting goals into one expert initially. Teachers can still interfere.
Illustrative example: Train rigor and format specialists before deciding how each teaches the student.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why train reward-specialized teachers and merge on student states?
SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation
Show answer and example
Answer: Separate rewards produce different strengths, then weighted distillation uses their distributions where students act. This isn't averaging rewards into one teacher or merging weights, and abilities can conflict. Local candidate evidence only.
Illustrative example: Calculation and formatting specialists can both score one student reasoning step.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How do SMOPD reward specialists differ from MOPD domain specialists?
SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation
Show answer and example
Answer: SMOPD varies reward priorities; MOPD often divides task domains. Both use multiple teachers, but expert grouping/evaluation and budgets differ.
Illustrative example: One domain can have several reward specialists, not one expert per dataset.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Which states should all SMOPD teacher KL terms align to?
SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation
Show answer and example
Answer: Score identical student prefixes, keeping weights/teacher targets frozen. Independent teacher generations do not align positionwise. No real local specialist training/merge.
Illustrative example: One teacher's step two and another's step four are not shared supervision states.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why specifically recover all-fail groups in RSTG?
Show answer and example
Answer: They lack relative reward signal and need teacher correction. All-success also has zero variance but no same failure need; consider variance and reward level.
Illustrative example: Don't call four correct replies a failure group just because their scores tie.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why ask the teacher only for zero-variance failures, not every tied group?
Show answer and example
Answer: All-fail normalization has no contrast, so teacher examples and high-entropy/gap distillation add signal. Tied successes don't need that intervention. Check outcome signs and teacher reliability.
Illustrative example: Request a verified demonstration for four wrong solutions, not four valid alternatives.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does RSTG token selection differ from imitating full teacher trajectories?
Show answer and example
Answer: Guide high entropy or large-gap positions more and add SFT on correct teacher traces. Selection avoids some irrelevant signals, but entropy/gap isn't ground-truth localization.
Illustrative example: Focus on uncertain reasoning rather than stable formatting.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Does implementing a negative-group gate complete RSTG?
Show answer and example
Answer: No. Execute local distillation and teacher-success supervision with provenance/masks. Local candidate objectives do not run large teacher generation or full budget.
Illustrative example: A need_teacher flag does not execute the teacher branch.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why rebuild VAD targets around the current student?
Show answer and example
Answer: Full teacher copying imports style/language bias. Add only visual-direction corrections around student to preserve other preferences; full privileged distribution is weak regularization.
Illustrative example: Transfer image color evidence without forcing all wording changes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why project teacher correction onto visual evidence and keep only aligned components?
Show answer and example
Answer: Matched views define a visual direction. Keep positive projection and anchor target to student; weakly regularize toward full teacher. This avoids broad stylistic imitation. Local reward axes don't reproduce visual training.
Illustrative example: Try to transfer support for a red object without importing unrelated teacher phrasing preferences.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does one-sided projection differ from replacing all correction with the visual direction?
Show answer and example
Answer: Keep only teacher correction positively aligned to evidence. Replacement ignores whether teacher supported that change. Projection constrains attribution, not correctness.
Illustrative example: If teacher correction opposes visual contrast, don't blindly inject it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
What should VAD evidence-present and evidence-removed branches share?
Show answer and example
Answer: Same frozen teacher, question and student prefix; only evidence view differs. Center logp and stabilize projection. Local process proxy lacks real crops/checkpoint/benchmarks.
Illustrative example: An unrelated text model would confound model architecture with evidence ablation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why distinguish reflective exploration from exposure bias in CRPO?
Contrastive Reinforced Policy Optimization via Privileged Self-Distillation
Show answer and example
Answer: Both can be uncertain. Reflection can repair reasoning; exposure bias can compound prefix errors. Group contrast treats them differently instead of rewarding entropy uniformly.
Illustrative example: Rechecking a branch is not the same as drifting after a wrong equation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why distinguish reflective and exposure-bias positions rather than reward all high entropy?
Contrastive Reinforced Policy Optimization via Privileged Self-Distillation
Show answer and example
Answer: Uncertainty can be useful exploration or failure from wrong prefixes. Contrast and privileged self-distillation handle them separately. Entropy alone isn't correctness; no complete reasoning training locally.
Illustrative example: High entropy during a useful recheck and a confused continuation call for different evidence-based updates.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does CRPO contrast differ from minimizing privileged teacher KL alone?
Contrastive Reinforced Policy Optimization via Privileged Self-Distillation
Show answer and example
Answer: Contrast creates relative competition between positive/negative positions; KL directly matches a target. Classification needs reliability; negatives don't eliminate bias automatically.
Illustrative example: Which negatives appear in the denominator changes the update.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Does local CRPO identify real reflection positions?
Contrastive Reinforced Policy Optimization via Privileged Self-Distillation
Show answer and example
Answer: No. Candidate objectives test masks/diagnostics only. Record position/group provenance, keep privilege out of rollout and never label test positions with gold then claim autonomy.
Illustrative example: Predefined masks don't demonstrate finding exposure bias in real text.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
How does Distilled RL use teacher distributions inside RL rather than standalone imitation?
Show answer and example
Answer: Teacher/old-student ratios reweight task advantages at token level for positives only. Task outcome picks reinforcement; teacher focuses credit.
Illustrative example: A verified solution's teacher-supported key step can receive extra weight.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why add teacher weights only to positive advantages?
Show answer and example
Answer: Verified positive samples already have outcome support; teacher ratio emphasizes useful tokens. Keep nonpositive learning unchanged. Clip and geometric-normalize weights without treating teacher preference as independent truth.
Illustrative example: Boost key steps in success; do not weaken negative updates just for teacher-approved wording.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does positive-only gating differ from weighting both signs?
Show answer and example
Answer: Nonpositive advantages keep weight one, preserving failure penalties. Two-sided weighting could weaken some negative feedback. The teacher does not relabel outcomes.
Illustrative example: Teacher preference for wording in a failed reply doesn't reduce failure penalty.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Why normalize clipped ratios by sequence geometric mean?
Show answer and example
Answer: Bound ratio scale and make weights relative within a valid-token sequence. Exclude padding and keep nonpositive branch one. Local mini-suite doesn't reproduce full generation/training.
Illustrative example: Including padding changes every weight even if its loss is masked.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why sample actively from reference in ARMOR?
Show answer and example
Answer: A passive penalty only acts on visited data and can't recover paths current policy never samples. Reference anchors restore actual training coverage.
Illustrative example: If student only takes a shortcut, old policy may still generate a verification route.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
How does active reference sampling differ from a KL penalty?
Show answer and example
Answer: KL constrains drift on sampled states without guaranteeing useful reference behaviors appear. Frozen reference rollouts add that coverage. Anchors may be bad; tune mix and don't generalize candidate proxies.
Illustrative example: Active replay reintroduces the old verification route that a passive penalty may never sample.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
Are reference rollouts equivalent to adding reference KL?
Show answer and example
Answer: No. Rollouts change visited states; KL regularizes distributions on existing states. Anchors may be stale or wrong and need evaluation.
Illustrative example: A missing old solution isn't practiced merely because KL is present.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
What is local ARMOR reference, and are its samples on-policy?
Show answer and example
Answer: The initialized candidate policy; log anchors separately from current student samples. No paper long-horizon pool; mixing tests data flow, not stability.
Illustrative example: Don't label an old-policy action as generated by the current policy.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why train independent domain teachers first in MOPD?
Show answer and example
Answer: Joint RL can couple domain gradients. Separate specialists optimize their domains before sharing dense student-prefix supervision. Integration still shares parameters and can interfere.
Illustrative example: Train math and code teachers separately, then guide one student.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why query multiple teachers on student traces rather than concatenate expert traces?
Show answer and example
Answer: Actual student prefixes reveal visited states. Matched teachers give dense supervision there, reducing trajectory mismatch. Supervision fusion isn't parameter averaging or guaranteed skill retention; local evidence is candidate-level.
Illustrative example: Both teachers can guide a mixed math/code intermediate state absent from their complete answers.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
Where do probability mixture and parameter averaging merge teachers?
Show answer and example
Answer: Probability mixtures form supervision at shared student states and require teacher inference. Parameter averaging directly combines comparable weights and can disrupt coordination. No superiority without matched experiments.
Illustrative example: Adding output distributions differs from averaging weight matrices.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
What must MOPD teacher weights and dimensions satisfy?
Show answer and example
Answer: Align vocabulary/candidates and normalize weights by the defined rule. Freeze teachers and update student only. Deterministic mini-suite teachers aren't trained specialists.
Illustrative example: Different vocabulary IDs don't align merely by renormalizing the sum.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why add a teacher-forced reference story to POLARIS policy stories?
POLARIS: Guiding Small Models to Write Long Stories
Show answer and example
Answer: Long story evaluation is sparse; a reference anchors quality under the same external rubric. It is additional training supervision, not test gold or a supplied copyrighted story.
Illustrative example: Practice with a reference story but generate independently at evaluation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why use policy-only statistics and gradually add reference advantage in POLARIS HRI?
POLARIS: Guiding Small Models to Write Long Stories
Show answer and example
Answer: Including reference in mean/SD would change every policy advantage. Use policy-only stats, scale reference gradually from zero, and apply identical rubric. Local component isn't full long-story training.
Illustrative example: Five policy stories determine baseline; reference computes its own advantage against them, not inside their average.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does gradual reference injection differ from equal mixing at the start?
POLARIS: Guiding Small Models to Write Long Stories
Show answer and example
Answer: Reference weight rises from zero to avoid early domination; policy stories alone determine mean/variance. Equal mixing changes strength and potentially baselines.
Illustrative example: Reference advantage weight at step zero must be zero.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
How does POLARIS avoid overflow without changing clipped objective?
POLARIS: Guiding Small Models to Write Long Stories
Show answer and example
Answer: Choose the exact branch by advantage sign in log-ratio space, clear padding/zero advantage first and exponentiate afterward. No extra clamp. Full story training was too costly; tensor gradients aren't capability.
Illustrative example: A capped branch doesn't need to compute a huge exp before taking min.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
What context-dependent learning can OPCD preserve?
Show answer and example
Answer: Prompt, retrieval or past experience vanishes when context clears. Distill a context-rich teacher into a context-free student on its own trajectories to internalize behavior.
Illustrative example: Learn a tool-use lesson that remains useful when the hint is absent next time.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why roll out without experience and score with an experience-aware teacher?
Show answer and example
Answer: Internalize external knowledge rather than preserve dependency. Student gets task only; teacher scores same prefix with experience; update by student-to-teacher reverse KL. Reliability matters and local checks don't reproduce long system prompts.
Illustrative example: Remove the instruction at evaluation to test whether the operation lesson lives in weights.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does OPCD differ from retrieving the same experience at inference?
Show answer and example
Answer: Retrieval depends on external context and read cost. Distillation reduces that dependence but can forget or overgeneralize. They can complement each other; compare empirically.
Illustrative example: Updated documents can be swapped immediately; older knowledge distilled into weights doesn't automatically update.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
What is OPCD reverse-KL direction and student input?
Show answer and example
Answer: KL from no-experience student to experience-conditioned teacher on identical student prefix. Experience stays teacher-only. Local conditional distributions aren't long prompts or paper environment.
Illustrative example: Swapping numerator to teacher changes KL direction.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Basics
Why can the same model be a more informed OPSD teacher?
Show answer and example
Answer: Teacher additionally sees a verified solution, while student gets problem only. The difference is information, not model size. It depends on reliable solutions.
Illustrative example: Judging with the conclusion known differs from solving it independently.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Core mechanism
Why must OPSD student rollouts not see the verified solution?
Show answer and example
Answer: The teacher uses it to score actual student prefixes; student stays in deployment conditions. Pointwise clipping bounds stylistic outliers. Giving student the answer bypasses learning. Local checks cover mechanism only.
Illustrative example: Teacher can evaluate a half-completed derivation knowing the answer; student cannot use that answer during independent generation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Method comparison
How does OPSD isolation differ from answer-conditioned SFT?
Show answer and example
Answer: Hide solution during student sampling and score its states with privileged teacher. Adding solution to student would teach unavailable deployment dependence. Full distribution distillation isn't imitation of one answer.
Illustrative example: A useful intermediate expression can appear for teacher but not as a shortcut in student prompt.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Teachers and correction: when extra context can help
Implementation details
Why clip OPSD pointwise rather than one response coefficient?
Show answer and example
Answer: Limit per-token divergence so unusual stylistic positions cannot dominate. Freeze teacher target and align masks. Local views aren't Qwen competition-math training.
Illustrative example: One anomalous format token needn't amplify the entire sequence's guidance.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Topic guide and reading order
Start with this question¶
How can a teacher with answers, visual evidence, or specialist hints help a student that cannot see them?
Takeaway¶
Teacher and student may be different conditional views of one model. OPSD gives the training teacher verified answers or reasoning while the student sees only deployable inputs. Multiple teachers and token routing select who supervises each position. Check how the extra information changes supervision, whether it suits the student, and whether it erodes already successful behavior.
Example¶
A student misreads a chart scale. A training teacher sees a crop or explanation and can give a correction distribution on the same student prefix. The student is updated with nonprivileged inputs. If math and code use separate experts, soft routing combines valid probabilities in a common vocabulary; averaging logits is not the same as mixing predictive distributions.
Common pitfall¶
Privileged references stay inside allowed training-teacher or scorer boundaries, not evaluated student inputs. A more confident teacher may be wrong; multiple teachers may not complement each other. Two-view, routing, and gradient components are not complete model distillation. Keep high-cost non-reproduction and local diagnostic boundaries visible.
Reading order¶
Read OPSD for shared parameters with different conditioning, Token-Level Teacher Selection for positionwise routing and mixtures, and MOPD for capability conflict and retention across teachers.
- OPSD · Original paper
- Compatibility Matters: A Framework for Token-Level Teacher Selection in On-Policy Distillation · Original paper
- MOPD · Original paper
Check your understanding¶
If the same model acts as teacher and student, is there necessarily no new signal?
Show suggested answer
No. Identical parameters can yield different conditional distributions under different evidence, contexts, or reasoning paths. That difference can train the student, but its benefit must be tested. Identical distributions or irrelevant style differences do not promise stable gains.