Which compute changes: question selection, shorter generation, branching, or conditional replay?¶

Updated 2026-10-27 · 38 topic entries / 6 representative comparisons

Data and budget methods intervene at different points in training. Curricula reassess current difficulty, LESSER reduces selection-feature cost, STOPD randomly generates less and corrects estimates, BranchOPD adds important branches, Lightning caches teacher scores, and TAR retrieves suppressed correct modes. One fewer teacher call does not mean the entire run costs less.

Mechanisms · Trade-offs · Cross-paper questions · Updates

How to read this card

The table compares representative methods. It is not a shared-benchmark leaderboard. Teaching examples are not measured results.

Mechanism comparison¶

Axes: Selection unit: question, token position, suffix, or historical trajectory · Extra costs: difficulty estimation, forward features, teacher scoring, and bank construction · Statistical contracts: inverse-probability weights, independent branches under a fixed policy, and cache consistency · Student changes: difficulty reassessment, current log-probability refresh, and gates for suppressed modes

Paper and source	Mechanism	Benefits and trade-offs
A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPO · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 边界与未完成项	Reassesses question difficulty from actual rollout success of the current policy and retains samples by curriculum stage.	Difficulty is not length or a permanent label. The local scaled curriculum and B5 control do not reproduce the full original progression.
LESSER: Post-Training Data Selection with Output-Layer Gradients · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Uses one forward pass to compute analytical output-layer gradient features instead of expensive per-example full-parameter gradients, without changing the selection rule.	The feature formula applies with an independent readout; it is not full-model gradient equivalence. Current evidence uses fixed random tensors, not downstream LLM training.
Does On-Policy Distillation Need to Complete Every Rollout? · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 结果、代码映射与边界	Samples anchors and sets block cutoffs independently before generation, weighting retained tokens by exact inclusion probability.	Unbiasedness concerns the gradient sum, not equality of normalized Adam updates. Full long rollouts were not reproduced; a real short-policy L1 component exists.
Branching On-Policy Distillation · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 代码映射与复现边界	Chooses forks by student uncertainty and teacher–student disagreement, samples independent suffixes from the same frozen student, and counts shared prefixes once.	Spends more budget at important states, not truncation or teacher forcing. The local short checkpoint lacks an executed equal-budget single-path baseline.
Lightning OPD · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Samples student trajectories and caches a consistent teacher's one-time scores for training.	Moves teacher cost into prefilling, not zero teacher cost or continual trajectory refresh. The local candidate cache does not reproduce Qwen token-distribution training.
Preserving Reasoning Diversity in Reinforcement Learning with Verifiable Rewards via Adaptive Replay · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 结果、代码映射与边界	Builds a diverse prototype bank from correct base trajectories and replays only trajectories and tokens that the current policy suppresses relative to the base.	Count bank generation and verification costs; the bank is finite. Full experiments were not reproduced and the random-model L1 diagnostic does not establish semantic diversity.

Trade-off analysis¶

Hard questions are not necessarily the most valuable for the current student¶

The curriculum uses current success rates, so an example can move into or out of a difficulty band after an update. LESSER approximates task-relevant gradient features rather than selecting by length or mistakes. Curriculum estimation costs rollouts, while LESSER needs candidate and target-query forwards. Compare total selection costs and never draw target queries from final test.

Sources: A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPO · LESSER: Post-Training Data Selection with Output-Layer Gradients

Generation, scoring, and replay are different costs¶

STOPD randomly truncates before generation and corrects by retention probability. BranchOPD expands student suffixes at a fork without duplicating prefix credit. Lightning caches teacher scores; TAR caches verified base behavior and activates it according to current suppression. They add or remove different work, so neither training-step count nor teacher calls during training alone suffice.

Sources: Does On-Policy Distillation Need to Complete Every Rollout? · Branching On-Policy Distillation · Lightning OPD · Preserving Reasoning Diversity in Reinforcement Learning with Verifiable Rewards via Adaptive Replay

Cross-paper questions¶

Try answering before expanding the explanation. Each question relates at least two papers.

If a curriculum finds a question too hard, should LESSER also discard it?¶

Show answer and example

Answer: Not necessarily. A curriculum selects by current policy success, while LESSER applies its selection rule to candidate and target-query output gradients. A difficult example may share target knowledge or be beyond reach; difficulty alone does not determine gradient relevance. Both incur selection costs.

Illustrative example: Teaching example: A domain question the student cannot yet solve may still share a target gradient direction. That neither guarantees value nor makes it zero.

Sources: A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPO · LESSER: Post-Training Data Selection with Output-Layer Gradients

STOPD and BranchOPD both change suffix budgets. Why not use the same early stop?¶

Show answer and example

Answer: STOPD randomly fixes a cutoff before generation and corrects retained tokens by inclusion probabilities. BranchOPD selects a fork and independently samples multiple continuations from the same frozen policy, counting the prefix once. One observes less and reweights; the other explores more at key states. Arbitrary post hoc truncation has neither contract.

Illustrative example: Teaching example: Randomly stopping at token 64 and branching into three continuations at token 64 are opposite uses of budget.

Sources: Does On-Policy Distillation Need to Complete Every Rollout? · Branching On-Policy Distillation

Why does TAR need current student probabilities while Lightning and TAR both reuse data?¶

Show answer and example

Answer: Lightning caches consistent teacher scores on initial student trajectories. TAR stores verified base behaviors and activates replay only where the current policy suppresses them relative to base, including token-level focus. A fixed bank does not fix weights, and zero teacher calls during training do not erase construction cost.

Illustrative example: Teaching example: A base solution still produced frequently can get zero TAR replay even though cached teacher scores still exist.

Sources: Lightning OPD · Preserving Reasoning Diversity in Reinforcement Learning with Verifiable Rewards via Adaptive Replay

Update log¶

Updates record new approaches, changed trade-offs, and follow-up evidence.

2026-10-27 · Initial edition: compares curriculum/LESSER selection, STOPD short generation, BranchOPD branching, Lightning teacher caching, and TAR conditional replay; 38 abstracts reviewed and six papers read closely.

Covers 38 topic entries; see the full list below.

All 38 topic entries

The full catalog follows. The main comparison covers the representative papers listed above.

Paper-by-paper questions · All deep dives