Trajectory distillation: separate sampler, proposal, and divergence direction¶

Updated 2026-11-04 · 42 topic entries / 8 representative comparisons

On-policy describes supervised states; FKL/RKL/JSD describe how distributions compare there. GKD and MiniLLM establish those axes, DrOPD and AP-OPD adjust correction location or strength, and Semi-OPD and learning-signal diagnostics distinguish offline reuse from residual loss. MPOPD adds a learnable proposal. Its geometric-mixture target differs from the student's teacher-directed RKL.

Mechanisms · Trade-offs · Cross-paper questions · Updates

How to read this card

The table compares representative methods. It is not a shared-benchmark leaderboard. Teaching examples are not measured results.

Mechanism comparison¶

Axes: State source: teacher demonstrations, current student, or fixed initial-student trajectories · Comparison geometry: FKL coverage, RKL mode seeking, and JSD routing · Local treatment: add a correction direction or rescale existing supervision · Evidence type: objective, caching contract, or learning-signal diagnostic

Paper and source	Mechanism	Benefits and trade-offs
SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 结果、负控与边界	Resamples current-draft private histories at fixed reference anchors, shares the trunk for parallel teacher scoring, and optimizes an acceptance-length surrogate using overlap prefix products.	Changes supervision-collection dependencies and the draft objective, not merely ordinary student RKL terminology. Full teacher mass and predecessor readout must remain intact. Full hybrid-target training/serving is not reproduced; tensor checks do not establish speedup.
GKD · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Lets the teacher supervise distributions at actual student-generated prefixes and mixes fixed with on-policy states through a student-data fraction.	Better state coverage does not prescribe a divergence. The local candidate support does not reproduce T5 sequence training.
MiniLLM · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Optimizes RKL on student generation with teacher-mixed sampling, baselines, and length treatment to stabilize estimates.	Favors teacher modes the student can represent and can weaken undercovered directions. Local evidence verifies candidate sampling and gradient direction.
Divergence-Routed On-Policy Distillation for Language Models · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 结果与验收	Keeps RKL everywhere and adds FKL only at high-JS positions, using a PPO surrogate over the union of candidates.	Supports teacher-probable candidates rarely sampled by the student, not a global switch to FKL. Full training was not reproduced; tiny online updates are L1 diagnostics.
Adaptive Pullback: Reweighting Supervision at Penalty Jumps in On-Policy Distillation · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 结果、代码映射与边界	Compares raw penalties in neighboring blocks and scales all signed supervision in a block after a spike.	No extra teacher forwards or sign change. Thresholds are not calibrated significance tests. Full runs were too costly and not reproduced.
When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 代码映射与复现边界	Freezes initial student trajectories and teacher scores but recomputes current student log-probabilities and advantages each step.	Avoids repeated collection without covering all new states. The short-budget checkpoint verifies training flow only.
Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 代码映射与复现边界	Tracks residual loss, pre-truncation gradient, two-fold occupancy, parameter displacement, and CKA to distinguish loss from usable signal.	A correlational diagnostic, not a new optimizer. The local three-step run does not recreate a long plateau or prove its cause.
Proposing Better Rollouts for On-Policy Distillation · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 高成本完整实验	Fits a separate proposal to the normalized geometric mixture of updated student and teacher, while weighting student RKL on proposal trajectories by a frozen single-step ratio lagged one token.	The ratio is not a cumulative product. Reset the proposal from the updated student each round. Complete sampling, LM copies, and optimizer orchestration are not implemented; tensor loss alone does not demonstrate on-policy training.

Trade-off analysis¶

Student trajectories can still support different learning directions¶

GKD supervises prefixes students actually visit but permits multiple divergences. MiniLLM chooses student-to-teacher RKL, which may weakly correct teacher options with very low student support. DrOPD retains RKL and adds FKL only on high-divergence positions over joint candidate support. State sampling, divergence direction, and updated candidates need separate checks.

Sources: GKD · MiniLLM · Divergence-Routed On-Policy Distillation for Language Models

Large loss, weak gradients, and spikes are not one fault¶

OPD Learning Signals logs losses and gradients jointly rather than treating residual discrepancy as strong learning potential. AP-OPD scales whole blocks after abrupt penalty rises without changing signs; it does not explain plateaus generally. Semi-OPD changes state reuse; fixed teacher labels do not freeze student-dependent advantages.

Sources: Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals · Adaptive Pullback: Reweighting Supervision at Penalty Jumps in On-Policy Distillation · When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better

A new sampler does not make the geometric mixture the student target¶

GKD supervises student prefixes and MiniLLM chooses student-to-teacher RKL. MPOPD keeps the teacher target but learns a separate proposal for reachability. The geometric mixture trains only the proposal. Correct sampling mismatch with the frozen old-student/old-proposal one-step ratio from the previous token, not a trajectory product. A real system must reset from the updated student each round; external logits alone do not establish trajectory provenance.

Sources: Proposing Better Rollouts for On-Policy Distillation · GKD · MiniLLM

A fixed outer reference can contain current-policy inner branches¶

GKD/MiniLLM supervise student-visited states; MPOPD adds an independently trained proposal to change reachability. SpecOPD instead targets speculative drafters: fixed reference anchors host freshly sampled inner branches, removing full-response verification dependencies. Fixed reference does not make these branches off-policy, but neither is this unrestricted student generation. E2E-TV needs full teacher mass and private predecessor histories. Passing its loss checks proves neither hybrid-target state sharing nor faster serving.

Sources: SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding · GKD · Proposing Better Rollouts for On-Policy Distillation

Cross-paper questions¶

Try answering before expanding the explanation. Each question relates at least two papers.

If a SpecOPD branch diverges after its reference anchor, should the teacher still score using the reference suffix?¶

Show answer and example

Answer: No. The teacher conditions on the reference prefix and the branch's sampled predecessors, not the reference continuation, future tokens, or siblings. The anchor predicts the first sample; predecessor rows predict later ones. Changing this history or reading the token's own row changes the method and can leak the answer.

Illustrative example: Teaching example: With reference a quiet and draft short evening, evening is predicted from a short, not a quiet, and must not be supplied as its own input first.

Sources: SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding · GKD

Why add FKL in DrOPD if GKD and MiniLLM already learn along student trajectories?¶

Show answer and example

Answer: Student trajectories determine prefixes only. MiniLLM RKL favors teacher modes the student supports, so low-probability teacher candidates can receive weak correction. DrOPD routes extra FKL at high-JS positions and retains teacher support. It does not change sampler or add unlimited FKL everywhere.

Illustrative example: Teaching example: A student considers only A at a visited mistaken prefix while a teacher favors B. Visiting the prefix does not guarantee a useful correction toward B.

Sources: GKD · MiniLLM · Divergence-Routed On-Policy Distillation for Language Models

Why cache Semi-OPD teacher scores but not teacher-minus-student advantages forever?¶

Show answer and example

Answer: Teacher configuration is fixed, but student parameters and log-probabilities change, so recompute differences. Stale advantages respond to old errors. Semi-OPD also fixes initial trajectories unlike continuously sampled GKD; neither implies all probabilities stay constant.

Illustrative example: Teaching example: An old teacher advantage can keep pushing a token the student has already learned.

Sources: When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better · GKD · MiniLLM

Are AP-OPD spike damping and DrOPD high-divergence FKL opposite corrections?¶

Show answer and example

Answer: They detect different things. AP-OPD checks neighboring raw penalty blocks and scales all signed coefficients; DrOPD detects JS divergence and undercoverage and adds a direction. Strength and composition differ.

Illustrative example: Teaching example: Persistent high divergence with no jump can trigger DrOPD but not AP-OPD; their gates are not complementary thresholds.

Sources: Adaptive Pullback: Reweighting Supervision at Penalty Jumps in On-Policy Distillation · Divergence-Routed On-Policy Distillation for Language Models · Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals

Should MPOPD distill the proposal's teacher–student mixture directly into the student?¶

Show answer and example

Answer: No. The mixture shapes proposal sampling; the student still learns RKL to the teacher. Student loss uses a frozen old-student versus old-proposal single-step ratio with one-token lag. This adds a proposal axis beyond sampled states and divergence direction.

Illustrative example: Illustration: A proposal can reach a teacher-supported prefix more often without replacing the teacher target by a compromise.

Sources: Proposing Better Rollouts for On-Policy Distillation · GKD · MiniLLM

Update log¶

Updates record new approaches, changed trade-offs, and follow-up evidence.

2026-11-04 · Reviewed SpecOPD, separating the fixed outer reference, current-draft inner branches, and E2E-TV. Adds collection dependencies beyond GKD/MPOPD, retaining full teacher mass, ancestry isolation, and predecessor readout without calling L1 tensor checks serving speedup.

Affected: SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding

2026-10-30 · Reviewed MPOPD v2, separating proposal, student RKL, teacher top-k support, and the frozen single-step ratio lagged one token. Adds a learnable sampler axis beyond GKD/MiniLLM and preserves prerequisites such as proposal reset.

Affected: Proposing Better Rollouts for On-Policy Distillation · GKD · MiniLLM

2026-10-27 · Initial edition: separates student state distributions and divergence, comparing GKD/MiniLLM, DrOPD/AP-OPD, Semi-OPD, and learning-signal diagnostics; 40 abstracts and six close reads.

Covers 40 topic entries; see the full list below.

All 42 topic entries

The full catalog follows. The main comparison covers the representative papers listed above.

Paper-by-paper questions · All deep dives