Data and budgets: where to spend the next unit of compute¶
简体中文 · English
LLM post-training / 152 questions · 38 papers
Try answering each question before opening the explanation. The bank grows as papers and questions are added.
Read this topic's comparisons, analysis, and cross-paper Q&A
152 questions · 38 papers
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why train an A2D question synthesizer before training student?
Ask to Distill: Learning to Synthesize Data for On-Policy Distillation
Show answer and example
Answer: Questions determine visited prefixes and usable guidance. Freeze scorer teacher/student, optimize question distribution, screen pool then distill from original student. Separate curriculum learning from student gains.
Illustrative example: Find learnable challenge problems without accumulating student updates while selecting them.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why require disagreement and teacher support within student reach?
Ask to Distill: Learning to Synthesize Data for On-Policy Distillation
Show answer and example
Answer: Disagreement alone can choose unlearnable problems, compatibility alone mastered ones. Reward their product using raw teacher mass, freeze scorers, then distill from original student.
Illustrative example: A teacher refinement near two known approaches has useful gap; mass almost entirely off-support should score low.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does A2D compatibility differ from pure disagreement reward?
Ask to Distill: Learning to Synthesize Data for On-Policy Distillation
Show answer and example
Answer: A big gap can mean impossible. Compatibility measures teacher mass on student support. Product balances room for learning and reachability rather than reward distance alone.
Illustrative example: A teacher-only path can have high disagreement but poor teachability.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Why not renormalize A2D compatibility on the candidate union?
Ask to Distill: Learning to Synthesize Data for On-Policy Distillation
Show answer and example
Answer: Use raw full-vocabulary teacher mass on student candidates; renormalization hides mass outside. Deduplicate across complete scoring batch, not one GRPO group; don't arbitrarily restandardize downstream advantages.
Illustrative example: If teacher mostly supports inaccessible tokens, local normalization overstates compatibility.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why collect actual correct base samples for TAR memory?
Preserving Reasoning Diversity in Reinforcement Learning with Verifiable Rewards via Adaptive Replay
Show answer and example
Answer: Recover modes base could truly generate and current training suppresses. Generate, verify with training-only checker and choose diverse prototypes. No correct samples means no bank; hand-written gold is not reachability evidence.
Illustrative example: A never-generated base solution cannot be labeled a forgotten skill.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why replay only correct traces suppressed relative to base?
Preserving Reasoning Diversity in Reinforcement Learning with Verifiable Rewards via Adaptive Replay
Show answer and example
Answer: Verified real base samples store diverse reachable solutions. Gate inactive if unchanged; if suppressed, concentrate on lowered tokens. Detached selection prevents optimization through the gate, and test labels are excluded.
Illustrative example: Restore disappearing algebra steps without repeatedly forcing a still-common diagram solution or inventing archived samples.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does adaptive TAR differ from repeated SFT on fixed archive?
Preserving Reasoning Diversity in Reinforcement Learning with Verifiable Rewards via Adaptive Replay
Show answer and example
Answer: Open the gate only when relative-to-base suppression occurs, focusing suppressed tokens. Fixed replay always supervises, potentially constraining new adaptation.
Illustrative example: Skip a still-common old solution until its likelihood falls.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
May TAR average only active gate trajectories?
Preserving Reasoning Diversity in Reinforcement Learning with Verifiable Rewards via Adaptive Replay
Show answer and example
Answer: No. Divide by all bank traces, including zero gates. Detach gate/token weights and use stable mixed precision. Active-only normalization amplifies rare activation.
Illustrative example: If just one trace is suppressed, active-only averaging makes it stronger than the bank mean.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why choose STOPD cutoff randomly before generating?
Does On-Policy Distillation Need to Complete Every Rollout?
Show answer and example
Answer: Independent advance sampling defines each position's inclusion probability and saves actual generation. Trimming after generation only saves supervision and can bias selection. Natural EOS remains valid.
Illustrative example: Choose a block cutoff before seeing the answer, not the most favorable length afterward.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why not normalize inverse-probability gradients by current random token count?
Does On-Policy Distillation Need to Complete Every Rollout?
Show answer and example
Answer: Late positions are less observable and inverse probabilities debias the sum. Dividing by a correlated random denominator needn't preserve expectation. Use previous weighted count, detached along with weights. Nonlinear optimizer steps still differ.
Illustrative example: Upweight seldom-observed later steps without claiming one short average equals full generation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
Does inverse-inclusion weighting make each Adam step identical?
Does On-Policy Distillation Need to Complete Every Rollout?
Show answer and example
Answer: No. The weighted gradient sum can match full-sum expectation, but normalization and nonlinear optimizer state matter. It is not per-batch update or final-training equality.
Illustrative example: Two short draws may average toward the full sum while giving different adaptive steps.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
How handle partial final blocks and STOPD normalizer?
Does On-Policy Distillation Need to Complete Every Rollout?
Show answer and example
Answer: Use exact coverage probabilities for the short tail and previous batch's weighted count as current normalizer. Keep begin/loss/finish lifecycle against reuse. Do not replace with current random count.
Illustrative example: A maximum length not divisible by block width still has exact tail probabilities.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why use a larger same-family SynthSelf rewriter?
Continually Post-training Small Models via Synthetic Self-distillation
Show answer and example
Answer: Small self-rewrites can distort content and original style may be difficult for students. A larger aligned rewriter supplies candidates verified by answer checks; shorter isn't automatically better.
Illustrative example: Convert a proof with skipped steps into learnable steps while retaining the correct result.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why rewrite with a larger related model and check the answer again?
Continually Post-training Small Models via Synthetic Self-distillation
Show answer and example
Answer: It can adapt training demonstrations to students but may introduce mistakes. Accept through answer checking and test fidelity/distribution requirements. Do not use evaluation answers to rewrite or condition the student.
Illustrative example: Verify that a simplified solution reaches the same conclusion; using test solutions invalidates generalization.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does answer rejection sampling differ from validating format?
Continually Post-training Small Models via Synthetic Self-distillation
Show answer and example
Answer: Valid structure does not ensure correctness. Check rewritten answers, then test distribution/fidelity constraints empirically. Same family guarantees neither.
Illustrative example: A syntactically correct JSON result with wrong arithmetic should be rejected.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Where may SynthSelf use gold?
Continually Post-training Small Models via Synthetic Self-distillation
Show answer and example
Answer: Offline training demonstrations can guide and validate rewrites. Never give evaluation answers to student or use them for curriculum. Real SFT remains necessary; preserve failed and repaired evidence separately.
Illustrative example: Text filtering without a parameter update isn't complete training.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why select high branching uncertainty among correct trajectories?
Show answer and example
Answer: Correct endings can hide fragile branch points. Monte Carlo continuations identify teachable states beyond binary original correctness.
Illustrative example: A proof with confusing mid-course formula options is useful despite a correct ending.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why profile teacher branch uncertainty, then select by student loss?
Show answer and example
Answer: Monte Carlo continuations identify fragile points on correct traces; shrink by teacher profile, then probe students and train on current high-loss items. Rerouting follows learning. Reduced student exposure is not end-to-end speedup.
Illustrative example: Keep a fragile proof first and switch after mastery; correctness alone cannot choose.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
What do teacher profiling and student NLL rerouting each solve?
Show answer and example
Answer: Teacher stage finds potentially difficult reasoning states; student probes find current gaps. Teacher difficulty isn't permanent student difficulty and isn't length.
Illustrative example: Move budget if the student already mastered a teacher-profiled difficult step.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Why doesn't reduced exposure imply equivalent total GPU acceleration?
Show answer and example
Answer: Exposure counts student training only, omitting real teacher profiling, probes and overhead. Log separately and keep evidence of generation.
Illustrative example: Expensive screening may outweigh two fewer training epochs.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why does GRPO curriculum difficulty change as policy improves?
Show answer and example
Answer: Measure difficulty through current sampled pass rate; learning and forgetting move items between bands. Static text length or old scores aren't the same.
Illustrative example: Yesterday's failed short prompt can become easy today.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why remeasure success instead of using fixed question length?
Show answer and example
Answer: Difficulty is policy- and budget-dependent, so real-rollout success bands shift with learning. Select phases on validation and retain hard-stage controls; length reward doesn't replace difficulty or justify test selection.
Illustrative example: A short arithmetic problem may be harder for current model than a familiar long problem.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
Why isn't online fixed-medium equivalent to progressing to hardest band?
Show answer and example
Answer: Fixed-medium preserves learnable success rates. Hardest progression can lose signal, and reported hardest-stage regression should remain visible.
Illustrative example: A curriculum crossing capability may yield only failures rather than progress.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
May phase selection use test scores or length reward to define difficulty?
Show answer and example
Answer: No. Select by validation, restore selected checkpoint and reestimate pass rates from actual current rollouts. Length reward is auxiliary, not difficulty. Static mock rates aren't online.
Illustrative example: Long output alone does not make a problem hard.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why sample extra suffixes at uncertain BranchOPD forks?
Branching On-Policy Distillation
Show answer and example
Answer: One rollout shows only one continuation. Frozen student branches expose multiple choices at the same state, focusing teacher supervision on disagreement rather than teacher-selected text.
Illustrative example: Try two continuations independently from the same proof prefix.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why count shared prefix only once?
Branching On-Policy Distillation
Show answer and example
Answer: Branching widens continuations, not earlier supervision. Select uncertain/disagreeing forks and sample independent frozen-student suffixes for teacher scores, without teacher forcing or gold filters.
Illustrative example: Count three shared steps once, then each of two suffixes separately.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does disagreement branching differ from retaining only correct branches?
Branching On-Policy Distillation
Show answer and example
Answer: It doesn't filter by outcome; success-only changes distribution after sampling. Preserve teacher scoring, termination supervision and matched cost/provenance.
Illustrative example: Failed branches still expose visited states; don't remove them for prettier scores.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Why not count shared BranchOPD prefix loss for every branch?
Branching On-Policy Distillation
Show answer and example
Answer: Added coverage is in suffixes. Deduplicate prefix and align valid positions. PBR supervises defined endings; boxed-then-EOS isn't arbitrary early stop.
Illustrative example: Five branches shouldn't multiply unrelated early-token learning fivefold.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why reevaluate old BoundaryEvo problems each round?
Show answer and example
Answer: Student capability changes; mastered boundary questions should leave replay and new ones enter. Curriculum coevolves rather than replay static hard problems.
Illustrative example: Move to new gaps after mastering an old proof.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why choose capability-boundary tasks then regenerate and verify teacher traces?
Show answer and example
Answer: Value depends on current expert–student gap and old tasks need rechecking. Generate and verify anew to avoid carrying stale wrong answers. Priming and verified supervision have different rules.
Illustrative example: Retire mastered decompositions and validate the teacher's new boundary solution instead of reusing an unchecked draft.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does capability priming differ from verified teacher supervision?
Show answer and example
Answer: Priming is small-scale behavioral warmup without correctness filtering. Later select questions, generate fresh teacher responses and independently validate. Don't present warmup as verified.
Illustrative example: Style imitation and checking a solution's correctness use different acceptance rules.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
What must BoundaryEvo record before SFT?
Show answer and example
Answer: Select tasks from current student/fixed expert/analyzer gap and quality, generate fresh verified teacher traces, and supervise only response tokens. No test references or old answers mislabeled fresh.
Illustrative example: A previous answer for another question breaks selection-to-supervision alignment.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why account for recoverability before pruning in ReCal?
ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery
Show answer and example
Answer: Severe pruning can induce poor student states that distillation struggles to repair. Probe damage informs channel reselection toward recoverable structures instead of relying solely on later training.
Illustrative example: Removing a key arithmetic path may not be fixed by simply distilling longer.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why measure pruning damage on a probe but reselect channels on the unpruned model?
ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery
Show answer and example
Answer: Probe reveals harmed teacher states; use their damage to reweight original activations. This can avoid poor student trajectories that are hard to repair. Preserve remaining vocabulary mass in damage KL.
Illustrative example: Raise calibration weight on carry states harmed by the probe instead of patching already removed channels arbitrarily.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does damage-weighted calibration differ from activation-size pruning?
ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery
Show answer and example
Answer: Probe teacher states for damage, then reweight unpruned model statistics. Activation magnitude alone ignores recovery. Match target width and restoration protocol for fair comparison.
Illustrative example: A rare but badly damaged state can gain calibration weight.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
May ReCal drop probability outside teacher top candidates and renormalize?
ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery
Show answer and example
Answer: No. Keep top events and remainder mass to estimate damage. Normalize response damage weights to unit scale; zero damage becomes uniform. Recompute selection from unpruned stats.
Illustrative example: Movement outside top tokens is invisible after local renormalization.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why switch PIVOT prompts to RL individually?
PIVOT: Perplexity-Informed KD-to-RL Transition Scheduling for Vertical-Domain Few-Shot Distillation
Show answer and example
Answer: Students are not equally ready on all prompts. Uniform switching removes guidance too early for hard cases. Rank teacher perplexity on student replies and expand RL among low-perplexity prompts.
Illustrative example: Let familiar tasks explore rewards while unfamiliar ones still distill.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why route low-perplexity problems to RL earlier?
PIVOT: Perplexity-Informed KD-to-RL Transition Scheduling for Vertical-Domain Few-Shot Distillation
Show answer and example
Answer: Teacher support on actual student responses indicates readiness for reward exploration. High perplexity retains supervision; aggregate by prompt and ramp RL share. This isn't a correctness or intrinsic difficulty test.
Illustrative example: Switch familiar teacher-supported steps sooner, while frequently divergent prompts continue distillation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
Why aren't teacher perplexity and verified correctness interchangeable?
PIVOT: Perplexity-Informed KD-to-RL Transition Scheduling for Vertical-Domain Few-Shot Distillation
Show answer and example
Answer: Perplexity measures teacher support for student wording, not task result. High support can be wrong and low support an effective novel route. It is a routing proxy.
Illustrative example: Teacher unfamiliarity with phrasing does not make the conclusion false.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
How turn PIVOT response scores into problem routing?
PIVOT: Perplexity-Informed KD-to-RL Transition Scheduling for Vertical-Domain Few-Shot Distillation
Show answer and example
Answer: Exponentiate each response's mean teacher negative logp, average per prompt, standardize within batch and choose lowest via cosine proportion. Preserve separate GRPO/OPD normalizations, not token-level routing.
Illustrative example: Don't decide a whole question from only its easiest completion.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why recursively decompose and compose MagicBeanStalk problems?
MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis
Show answer and example
Answer: All-fail hard tasks provide no advantage. Effective subproblem examples reveal dependency blueprints to synthesize new leaves and compose tasks with mixed current-policy outcomes. Valid format isn't efficacy.
Illustrative example: Practice manageable parts and recombine them without losing key dependencies.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why can't subproblem effectivity replace final Goldilocks selection?
MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis
Show answer and example
Answer: Parent improvement checks decomposition on one learner. Final task must not be all-correct or all-wrong under target budget and needs normally completed correct trace. Local teacher probes don't complete recursive construction.
Illustrative example: Helpful subproblems for one student do not make a composed task learnable if target never solves it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
What do subproblem effectivity and target-policy Goldilocks each verify?
MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis
Show answer and example
Answer: Effectivity asks if subexamples improve a student's parent success. Goldilocks requires mixed outcomes and at least one normally terminated success under target budget. One doesn't replace the other.
Illustrative example: An auxiliary learner may improve while target still fails every final sample.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Can a well-formed local MBS problem be considered accepted?
MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis
Show answer and example
Answer: No. Need complete correct solution, independent validation, blueprint/novelty and effectivity/Goldilocks checks. Retain real construction failures; isolated teacher diagnostics aren't acceptance.
Illustrative example: Valid JSON and dependencies do not rescue an unsolvable task.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why drop some high-probability positive-advantage replies in GRPODropout?
GRPODropout: Less is More for Online Reinforcement Learning Rollouts
Show answer and example
Answer: They can oversharpen common modes. Selectively drop by entropy contribution while retaining failures. It isn't neuron dropout and doesn't refund generated rollouts.
Illustrative example: Reinforce fewer repeats of a common successful template.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why filter high-probability positives but retain all nonpositives?
GRPODropout: Less is More for Online Reinforcement Learning Rollouts
Show answer and example
Answer: Dominant successes can further reduce entropy; select by entropy contribution, not random removal. Preserve negative learning and recenter with old probabilities. Generation cost is already paid.
Illustrative example: Downweight repeated successful templates without turning failures into omitted feedback.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does response dropout differ in cost from generating fewer replies?
GRPODropout: Less is More for Online Reinforcement Learning Rollouts
Show answer and example
Answer: Generate full group then filter updates; fewer samples would reduce generation and coverage, changing estimator. Less backward work isn't equal end-to-end savings.
Illustrative example: Using six of eight generated replies does not cost only six generations.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Can GRPODropout keep original centered advantages after filtering?
GRPODropout: Less is More for Online Reinforcement Learning Rollouts
Show answer and example
Answer: No. Restore all nonpositive samples and recenter retained ones by old-policy probability. Dropping either condition changes feedback/baseline.
Illustrative example: Removing strong positives shifts the reference and requires recentering.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why multiply DIAL-OPD log gaps by logarithmic probability mean?
DIAL-OPD: Learning More from Fewer Tokens in On-Policy Distillation
Show answer and example
Answer: Tokens near impossible under both models can have large log differences. Scaling downweights such noise and prioritizes supported disagreements. Selection is not verifier truth.
Illustrative example: A bizarre word differing tenfold may matter less than a small gap on a key digit.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why not select solely by largest teacher–student logp gaps?
DIAL-OPD: Learning More from Fewer Tokens in On-Policy Distillation
Show answer and example
Answer: Huge gaps can occur in negligible mass. Logarithmic probability mean rescales priorities. Keep masked tokens as context; sparse supervision does not reduce generation or prove errors.
Illustrative example: An unlikely odd word should not drown out disagreement on a realistic key step.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does sparse DIAL supervision differ from deleting unselected text?
DIAL-OPD: Learning More from Fewer Tokens in On-Policy Distillation
Show answer and example
Answer: Unselected tokens stay as context but have masked losses. Removing text changes later prefixes and teacher scores.
Illustrative example: Keep preceding arithmetic/connectives even when not training their tokens.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
How handle equal p and q in DIAL's logarithmic mean?
DIAL-OPD: Learning More from Fewer Tokens in On-Policy Distillation
Show answer and example
Answer: Use its continuous limit. Choose each response's own valid-position quota, detach advantages and differentiate final student logp only. Global ranking can let long answers dominate.
Illustrative example: Equal probabilities have a finite limit, not NaN or an arbitrary large score.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why use output-layer gradients as LESSER selection features?
LESSER: Post-Training Data Selection with Output-Layer Gradients
Show answer and example
Answer: Full per-example backprop is costly. Independent readout gradient follows analytically from prediction residual and final hidden state in one forward, then project for the existing selector.
Illustrative example: Keep query similarity selection but replace expensive gradient features.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why isn't one-forward analytic output gradient an exact full-model substitute?
LESSER: Post-Training Data Selection with Output-Layer Gradients
Show answer and example
Answer: The outer product is exact for an independent readout; shared bilateral projection compresses it for existing selection. Tied embeddings can introduce indirect gradients. Use validation proxies and isolate test. This is not all-layer gradient equality.
Illustrative example: Select training examples using readout gradients of validation queries, not test examples.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
Is LESSER fully equivalent to full-model gradient similarity?
LESSER: Post-Training Data Selection with Output-Layer Gradients
Show answer and example
Answer: No. Exactness applies to an independent output layer; tied embeddings add hidden-state paths. Projection also approximates geometry and downstream performance needs testing.
Illustrative example: Similar final-layer directions do not guarantee similar middle-layer updates.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Why share random projections between LESSER candidates and queries?
LESSER: Post-Training Data Selection with Output-Layer Gradients
Show answer and example
Answer: Only shared coordinates support comparison. Project both residual and hidden sides without materializing the full outer product. Use validation query proxies, never test selection; local random tensors are diagnostics.
Illustrative example: Independent projection draws per sample destroy comparability.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why doesn't FTW need same-prompt groups or a critic?
Show answer and example
Answer: Stateful tool trajectories are expensive. Repeatedly draw minibatches from recent real FIFO traces, imitate within-batch elites and apply KL. Replay ranks replace some group sampling/value needs.
Illustrative example: One trace per task can still participate in a pool minibatch tournament.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why repeatedly select small-batch elites rather than pool top-K?
Show answer and example
Answer: Random minibatches followed by local top reward yield smoothed occupancy, with repetition weighting. Hard global elites are a special case and may drop useful runners-up. No same-prompt groups/critic needed, but KL remains and local tool gains aren't proven.
Illustrative example: Strongest often wins but a runner-up can win some batches; don't deduplicate then equally weight them.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
Why aren't repeated minibatch elites equivalent to one global top-K?
Show answer and example
Answer: Repeated batches create soft rank occupancy; near-elites sometimes win. Global hard truncation retains only top ranks. Repeated selection counts are weights, so deduplication changes the objective.
Illustrative example: A second-tier trajectory can win a batch without the strongest opponents.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
What are FTW limits K=C and C=N?
Show answer and example
Answer: K=C keeps the whole sampled minibatch and yields uniform sampling. C=N draws the whole pool and yields global elites. Verify normalized distribution and preserve repeated selections; bandits aren't tool-world results.
Illustrative example: If K=C still favors highest rank, the implementation misses the limit.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why represent DARS progress as a dependency predicate graph?
Dependency-Aware Reward Shaping for Agentic Reinforcement Learning
Show answer and example
Answer: Completed steps can rely on later-invalid conditions. Graph validation, invalidation and repair update related credit without clearing independent branches.
Illustrative example: Canceling a booking can affect itinerary but not unrelated document preparation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why decay nearby credit rather than reset every completed step?
Dependency-Aware Reward Shaping for Agentic Reinforcement Learning
Show answer and example
Answer: Invalid prerequisites affect dependent branches, not independent work. Graph-distance decay and potential differences encode breakage/repair. Valid predicates and edges are prerequisites; local transitions aren't agent gains.
Illustrative example: Ticket failure weakens hotel credit but not independent reimbursement materials.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does dependency-distance decay differ from global failure penalty?
Dependency-Aware Reward Shaping for Agentic Reinforcement Learning
Show answer and example
Answer: Decay depends on relation to the broken prerequisite; global penalty ignores it. Wrong graph or predicates still misassign credit, so validate them.
Illustrative example: Flight-dependent hotels are affected, not unrelated account verification.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Does DARS repair change only immediate reward?
Dependency-Aware Reward Shaping for Agentic Reinforcement Learning
Show answer and example
Answer: No. Update dependency state/potential and derive reward from differences. Otherwise later credit uses stale breakage. Local state kernel doesn't train paper model; upstream interfaces aren't released training code.
Illustrative example: Repaired flights restore downstream state rather than add an isolated bonus.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why lower GATS teacher weight as the student closes the gap?
Guide, Then Let Go: Gap-Adaptive Teacher Scheduling for Sparse-Reward Agentic RL
Show answer and example
Answer: Teacher gives direction early under sparse rewards, but near a fixed reference it can restrict independent exploration. Permanently withdraw at the target and continue reward training.
Illustrative example: After reaching teacher-level success, continued imitation may hinder improvements.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why lag GATS success and permanently withdraw after catching teacher?
Guide, Then Let Go: Gap-Adaptive Teacher Scheduling for Sparse-Reward Agentic RL
Show answer and example
Answer: Lag avoids same-batch feedback and fixed benchmark stabilizes comparison. Linearly reduce distillation as gap closes, then keep teacher off despite fluctuations. Noise remains and only state machine was validated.
Illustrative example: An improvement affects guidance next batch; reaching benchmark isn't a reversible toggle.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does permanent withdrawal differ from switching per batch?
Guide, Then Let Go: Gap-Adaptive Teacher Scheduling for Sparse-Reward Agentic RL
Show answer and example
Answer: Permanent state avoids oscillatory recalls; dynamic switching can restore help after a drop. Noisy pass rates mean reaching target once isn't universal superiority.
Illustrative example: A slight later decline won't restart teacher calls by design.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Why use one-step-lagged EMA and fixed teacher tail success?
Guide, Then Let Go: Gap-Adaptive Teacher Scheduling for Sparse-Reward Agentic RL
Show answer and example
Answer: Current rewards cannot alter their own schedule. Fixed teacher baseline defines gap; handle zero baseline. Only schedule kernel, not Qwen agent RL.
Illustrative example: Log current batch then let next batch read updated EMA.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why reuse historical self-rollout in ROSS without imitating all of it?
ROSS: Relearning from Self-Generated Rollouts through Selective Supervision
Show answer and example
Answer: Old traces contain mistakes/redundancy. Full context conditions later behavior; selective continuation labels retain only valuable segments. Visible text need not be target.
Illustrative example: Read the failure history but supervise only its selected correction.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why retain full context but supervise selected continuation only?
ROSS: Relearning from Self-Generated Rollouts through Selective Supervision
Show answer and example
Answer: History determines what state the continuation addresses. Masked supervision avoids imitating mistakes/redundancy while keeping context visible. Fixed local masks do not implement the paper selector.
Illustrative example: An early error can remain visible while only a later successful repair is learned.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does ROSS differ from deleting unselected spans before SFT?
ROSS: Relearning from Self-Generated Rollouts through Selective Supervision
Show answer and example
Answer: Deletion changes conditional state, while full context plus masked loss preserves the trace. Selector quality still matters; mask correctness is not selector implementation.
Illustrative example: Removing a wrong prefix can change meaning/likelihood of a corrective sentence.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Is the local ROSS selector learned?
ROSS: Relearning from Self-Generated Rollouts through Selective Supervision
Show answer and example
Answer: No. A fixture mask verifies full-context/selective-loss boundaries, not autonomous selection of useful historical steps.
Illustrative example: Marking the last two sentences manually tests adherence, not discovery.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why turn environment feedback into retrospection before ROFT learning?
Shockingly Simple Self-retrospection Improves Agentic Models Without RL
Show answer and example
Answer: Terminal reward rarely expresses reusable lessons. Text reflections summarize useful experience from successes/failures. The reflection is the target, not every historical action.
Illustrative example: A permission mistake can teach checking access before next time.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why supervise only retrospection tokens, not actions or feedback?
Shockingly Simple Self-retrospection Improves Agentic Models Without RL
Show answer and example
Answer: History conditions the lesson, but the target is the later reflection. Separate input and loss masks and globally average reflection tokens. Local fixtures/registry aren't a complete agent/Evolve loop.
Illustrative example: Read a bad call and error, but learn the parameter-check lesson, not the bad call or tool error text.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does ROFT differ from cloning original actions?
Shockingly Simple Self-retrospection Improves Agentic Models Without RL
Show answer and example
Answer: Cloning repeats recorded behavior; reflection supervision can teach avoiding it. Different labels and reflection quality still matters.
Illustrative example: After deleting a wrong file, learn to check paths rather than repeat deletion.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
How normalize losses across differently sized ROFT retrospections?
Shockingly Simple Self-retrospection Improves Agentic Models Without RL
Show answer and example
Answer: Average all valid reflection tokens globally. Keep tasks/actions/observations/feedback as context but mask their loss. Per-reflection means change length weights. No full agent data/Evolve loop.
Illustrative example: Short and long reflections contribute different total mass under token vs example means.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why make RetireOPD teachers skill-specific and retirable?
RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
Show answer and example
Answer: Mastery varies. Retire when student approaches skill teacher and distribution improvement stalls, then explore with RL. Fixed uniform budgets can overteach known skills and underteach weak ones.
Illustrative example: Search and output format can retain different guidance.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why not retire every teacher once overall student success is close?
RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
Show answer and example
Answer: Each skill checks both outcome closeness and gap no longer shrinking. One criterion risks early withdrawal. Local mechanism checks don't reproduce paper teacher training.
Illustrative example: Search may qualify while formatting still lags; one total score can't retire both.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
Why use both success and distribution gap for retirement?
RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
Show answer and example
Answer: Similar success can leave valuable behavior differences. A stalled gap can occur despite low success. Joint thresholds estimate remaining teaching value, not universal teacher surpassing.
Illustrative example: Matching task scores alone don't show tool choices stopped improving.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Do local retirement conditions show actual decoupled teachers were trained?
RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
Show answer and example
Answer: No. Only retirement state and student update kernel. Log skill identity, success, gap trends and state; registry labels aren't multi-teacher training.
Illustrative example: A synthetic retired state tests a branch, not genuine replacement.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why isn't trajectory learnability just success reward?
Trajectory Learnability for Offline On-Policy Distillation
Show answer and example
Answer: Success describes outcome, whereas current student mastery determines learning value. Token likelihood changes between successful reference and current policy reweight offline traces for remaining opportunity.
Illustrative example: Two correct solutions can differ because student already knows one and misses the other's key step.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why compare token likelihoods rather than rank offline traces by reward?
Trajectory Learnability for Offline On-Policy Distillation
Show answer and example
Answer: Success alone doesn't locate learning opportunities. Train reference on successes and compare likelihoods to current policy to estimate learnability. Depends on reference and offline coverage; local kernel only.
Illustrative example: Reweight two correct traces differently when one is mastered and the other has a difficult turning point.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does offline reweighting differ from collecting fresh on-policy data?
Trajectory Learnability for Offline On-Policy Distillation
Show answer and example
Answer: Weight allocates budget inside existing coverage. New sampling reaches new states. OPD in the name doesn't make replay current-policy generation.
Illustrative example: Reweighting a cache cannot create an absent solution.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Does local learnability aggregation include full GPU training?
Trajectory Learnability for Offline On-Policy Distillation
Show answer and example
Answer: No. It implements token aggregation and trajectory weighting. Score identical text/prefix under both versions and record them; don't subtract different responses.
Illustrative example: Old model scoring A and current model scoring B confounds text change.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why grow draft heads during RL in GrowMTP?
GrowMTP: Can RL Grow Its Own Draft Head?
Show answer and example
Answer: Main policy evolves, making static drafters stale. Train multi-token heads with rollout acceptance to follow policy. Faster drafting isn't task capability improvement.
Illustrative example: If policy learns new phrasing, drafter should propose it to retain acceptance.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why train by actual accepted depth and detach from backbone gradients?
GrowMTP: Can RL Grow Its Own Draft Head?
Show answer and example
Answer: Later drafts only contribute after earlier acceptance, so observed depth reflects use. Decoupling protects reward learning. Hardware and verification costs determine actual speed; local loss isn't acceleration or capability evidence.
Illustrative example: A third draft token isn't as reachable as first, and improving it doesn't prove stronger policy.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does acceptance-depth learning differ from equal future-head supervision?
GrowMTP: Can RL Grow Its Own Draft Head?
Show answer and example
Answer: A later head helps only after earlier proposals are accepted. Depth coupling reflects use, unlike equal head contributions. Still verify with target.
Illustrative example: Head three saves a step only if the first two proposals pass.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Why decouple GrowMTP gradients, and was speed measured?
GrowMTP: Can RL Grow Its Own Draft Head?
Show answer and example
Answer: Keep draft optimization from inadvertently changing main RL objective. Local code tests acceptance-depth loss only, without measured GPU throughput. End-to-end speed needs timing.
Illustrative example: More accurate drafting may also add compute.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why restart STRIDE-OPD from the weakest correct turn?
Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation
Show answer and example
Answer: A cached learnable state focuses budget on barely mastered turns rather than regenerating familiar prefix. Correct qualification ensures a valid starting point.
Illustrative example: Retry from a second turn that only just succeeds once the first is stable.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why pair early stopping and restart from weakest correct turn?
Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation
Show answer and example
Answer: Teacher cumulative confidence avoids spending on low-value continuations. A weak but correct cached turn reassigns budget to practice. These are distinct choices, not jumps to gold. No real-model throughput validation.
Illustrative example: Stop an uninformative continuation, then retry from an earlier barely successful turn rather than repeatedly replay the familiar beginning.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does early stopping differ from uniform truncation?
Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation
Show answer and example
Answer: Teacher cumulative confidence triggers a state-dependent stop paired with prefix restart. A fixed length limit changes coverage differently; shortness doesn't guarantee completion or savings.
Illustrative example: A solved trace and exploratory trace need not stop at the same token count.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Can STRIDE restore any environment from cached prefix text?
Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation
Show answer and example
Answer: Align cached prefix with executed actions and observations. Text alone may omit environment state. Local state machine isn't LLM rollout throughput.
Illustrative example: Do not duplicate a tool side effect on restore and call it the same state.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
What dependency does data-free OPD reduce?
Data-free On-policy Distillation
Show answer and example
Answer: Teacher-generated prompts reduce collection of a separate post-training problem bank. Teacher knowledge and sampling still provide information.
Illustrative example: Teacher can create exercises without a human bank using existing learned skills.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why isn't self-generated data-free OPD independent of prior knowledge?
Data-free On-policy Distillation
Show answer and example
Answer: Teacher parameters, sampling and filters determine curriculum even without a separate task bank. Generated tasks can be repetitive/unreliable. Local diagnostics have not generated a natural-language dataset.
Illustrative example: Generating code exercises avoids human curation but repeated loop problems do not teach omitted concurrency.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does generated curriculum trade off with fixed public tasks?
Data-free On-policy Distillation
Show answer and example
Answer: Generated tasks can target teacher skills but repeat or omit topics. Fixed banks have auditable coverage and collection cost. Validate quality; linguistic format isn't task validity.
Illustrative example: A teacher repeating linear equations does not cover geometry automatically.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Has local DF-OPD generated a real natural-language training set?
Data-free On-policy Distillation
Show answer and example
Answer: No. Only policy-feature diagnostics for generated questions. Need actual generation, filtering and distillation provenance, not changing candidate indices.
Illustrative example: A feature ID isn't a verified math problem.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why constrain LOCUS updates to task-related low rank?
LOCUS: Task-Aware Low-Rank Post-Training for Token-Efficient Language Generation
Show answer and example
Answer: Compact directions can improve token efficiency while limiting unrelated drift. If subspace misses key skills, performance can suffer.
Illustrative example: Removing redundant language may not require unrestricted changes to all parameters.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why can task-related low-rank updates shorten responses without truncation?
LOCUS: Task-Aware Low-Rank Post-Training for Token-Efficient Language Generation
Show answer and example
Answer: Learning operates on parameter directions and aims to preserve task-relevant changes. Truncation deletes content afterward, potentially essential. Validate subspace, correctness and efficiency; proxy SVD isn't LLM evidence.
Illustrative example: Learn to omit redundant checks, not cut the second half of every proof.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does low-rank learning differ from truncating output?
LOCUS: Task-Aware Low-Rank Post-Training for Token-Efficient Language Generation
Show answer and example
Answer: Train more concise behavior rather than deleting already generated steps. Check correctness alongside length, not shorter alone.
Illustrative example: Omitting repetition and cutting a proof's conclusion both shorten text but differ in quality.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Does local LOCUS SVD train a real language model?
LOCUS: Task-Aware Low-Rank Post-Training for Token-Efficient Language Generation
Show answer and example
Answer: No. It projects policy gradients in a feature subspace. Check rank, orthogonality and gradient paths. A random low-rank matrix isn't learned semantic direction.
Illustrative example: Projection success doesn't establish coverage of core reasoning.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why combine multiple signals for adaptive OPD gating?
Show answer and example
Answer: Entropy, uncertainty, reference drift and teacher/student gap capture different conditions. Combine them to allocate per-token forward/reverse KL; signal utility must be validated.
Illustrative example: Teacher confident but far from reference differs from uncertain but close to student.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why mix FKL and RKL per token rather than select one direction per answer?
Show answer and example
Answer: Uncertainty, gaps and reference drift vary by token. Allocate coverage and concentration accordingly. Gate is a supervision heuristic, not truth, and local tests don't show stable superiority.
Illustrative example: Preserve multiple options at a pivotal branch and use concentrated guidance for settled format.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does per-token FKL/RKL differ from one ratio per reply?
Show answer and example
Answer: Token gates vary coverage vs concentration by position; fixed mixture applies everywhere. Flexibility increases tuning/overfit risk; diagnostics aren't consistent wins.
Illustrative example: A reasoning branch and predictable format token need not receive the same mix.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Can local four-signal gates qualify for capability promotion?
Show answer and example
Answer: No. Deterministic mechanism tests don't establish full-model effects or fair compute. Specify direction, signal versions and gradients; registry changes are not training.
Illustrative example: One improved small-candidate score doesn't generalize to open language.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why study supervision at one or two positions in Sparse OPD?
Extremely Sparse Supervision Incentivizes Reasoning Ability
Show answer and example
Answer: Dense supervision may be redundant. Select important positive teacher–student differences to test concentration of useful information. A particular experiment doesn't prove one token suffices universally.
Illustrative example: Correcting one operator can matter more than imitating many connectives.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why retain full context while supervising only a few largest positive gaps?
Extremely Sparse Supervision Incentivizes Reasoning Ability
Show answer and example
Answer: Teacher distributions depend on actual preceding reasoning. Sparse gradients don't remove conditioning. Limited-position gains are setup-specific, and no local full-parameter distillation was run.
Illustrative example: Score one key correction with prior calculations intact; deleting the prefix changes the task.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
Why aren't sparse loss and shorter generation the same efficiency improvement?
Extremely Sparse Supervision Incentivizes Reasoning Ability
Show answer and example
Answer: Sparse loss reduces supervised positions while keeping full trajectories and often scores them to find key tokens. Shorter generation changes cost and state coverage.
Illustrative example: Generating a thousand tokens and training one still pays for all generation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
How distinguish unselected-token gradients from context?
Extremely Sparse Supervision Incentivizes Reasoning Ability
Show answer and example
Answer: Unselected positions have no direct loss but stay as conditioning; shared weights may still affect them. Local candidate tests aren't full Qwen/Llama training.
Illustrative example: Omitting step two's loss doesn't allow deleting it before scoring step three.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why use fast and slow draft adaptation in SpecRoll?
SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts
Show answer and example
Answer: RL changes target policy. Lightweight hidden corrections handle short drift; persistent degradation triggers future-head training. Both improve proposals without replacing target.
Illustrative example: Quickly adjust a transient acceptance drop, retrain the head only if it persists.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why can't fast correction or slow draft-head training replace exact verification?
SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts
Show answer and example
Answer: They improve proposals as policy evolves; exact rejection/acceptance preserves target distribution. Skipping it changes RL data even with a strong drafter. No local large-model throughput evidence.
Illustrative example: Adapt the drafter after lag, but still verify every proposal.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
Does changing the SpecRoll draft change GRPO target?
SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts
Show answer and example
Answer: No. Exact target verification preserves sampling distribution, so GRPO stays unchanged. Skipping verification would change behavior data.
Illustrative example: Frequent correct drafts don't make their probabilities target policy probabilities.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Does local SpecRoll registration prove large-model RL throughput?
SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts
Show answer and example
Answer: No. Only states and selection paths are checked. Real work needs acceptance/rejection residuals, scheduling and compute measurements; listing isn't execution.
Illustrative example: A listed candidate does not demonstrate complete GPU training.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why learn recoverability rather than just uncertainty in RAIL?
Optimizing What Policies Learn From: Recoverability-Aware Rollout Intervention Learning
Show answer and example
Answer: An uncertain state may be irreparable. Intervene based on expected recovery net of cost and condition on location/action, instead of uniformly spending extra.
Illustrative example: One missing tool retry and a fundamentally wrong premise have different recovery value.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why optimize intervention location/method with cost instead of oversampling uncertain states?
Optimizing What Policies Learn From: Recoverability-Aware Rollout Intervention Learning
Show answer and example
Answer: Additional budget helps only recoverable states. Learn intervention-specific net gains from auditable shadow/live feedback as policy changes. Never use gold to decide; no local full environment.
Illustrative example: One more tool call can rescue a trace while no reasonable budget repairs a false premise; entropy alone can't tell.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does shadow-to-live evaluation differ from gold-based intervention choice?
Optimizing What Policies Learn From: Recoverability-Aware Rollout Intervention Learning
Show answer and example
Answer: Use permitted probes and actual feedback to learn recovery value. Gold would be an unavailable oracle. Deployment state must exclude training-only diagnosis.
Illustrative example: Restarting from a known solution plan is an oracle diagnostic, not a learned controller.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
What does the local RAIL path establish?
Optimizing What Policies Learn From: Recoverability-Aware Rollout Intervention Learning
Show answer and example
Answer: Reference intervention selection and cost objective, not the paper's LLMs, private environment or judges. Execute and evaluate real recoveries before capability/budget claims.
Illustrative example: Argmax over random recovery scores does not measure actual trajectory repair.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why use pass@k to probe CoBA-RL boundaries before teacher help?
Show answer and example
Answer: Pass@1 can increase by concentrating old successes. Larger-budget coverage finds currently difficult reachable edges, where teacher guidance supplies new routes for RL consolidation.
Illustrative example: Occasional successes becoming frequent differs from learning a previously unseen solution.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why inspect both pass@1 and larger pass@k for capability boundaries?
Show answer and example
Answer: First-hit gain may just popularize known paths. Multiple samples probe coverage, then teacher and RL target boundaries. Finite budgets still miss rare paths; local small sampling does not reproduce high-k expansion.
Illustrative example: Measure whether once-never-solved prompts become reachable with broader sampling, counting teacher cost.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does a boundary curriculum differ from ranking fixed textual difficulty?
Show answer and example
Answer: Reestimate from current policy samples; textual complexity doesn't track mastery. Finite k can miss rare successes, so boundary depends on budget and isn't impossibility.
Illustrative example: A long problem may leave the boundary after learning a new technique.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Does local pass@8 establish paper pass@256 gains?
Show answer and example
Answer: No. Teacher/student scales and sampling budgets differ. Genuine generation, triggers and cost curves support scaled mechanisms only.
Illustrative example: Failure in eight draws doesn't imply failure after over 200 draws.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why emphasize equal-label-budget stages in the sparse-to-dense reward principle?
Show answer and example
Answer: Dense teachers have different supervision costs. Match label budgets and stages to understand ordering. Contribution is an empirical four-stage comparison, not another KL function.
Illustrative example: More teacher calls can improve scores without proving dense objectives caused the gain.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why isn't the principle just sequencing GRPO and KL functions?
Show answer and example
Answer: The paper tests four stages under equal label budgets and distinguishes forward KL on teacher-visited states from reverse KL on student states. Operators alone don't reproduce it. No local complete training.
Illustrative example: The same KL name on a teacher solution and a student error refers to different supervision.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
Is the same KL direction equivalent on teacher and student occupancy?
Show answer and example
Answer: No. Teacher ideal and student actual prefixes visit different errors. State source and divergence together define supervision.
Illustrative example: A KL on a correct proof and on a wrong intermediate state teaches different recovery.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Do existing GRPO and OPD operators reproduce the four stages?
Show answer and example
Answer: No. Literature only; no full scheduling, real frozen teacher or equal-budget runs. Shared tokenizer and other prerequisites still matter. Stage labels aren't execution.
Illustrative example: Listing four configuration stages does not validate their data flow.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why cache Lightning OPD teacher scores before updates?
Show answer and example
Answer: Online teacher inference can bottleneck training. Score initial student traces once and reuse labels. This front-loads cost rather than eliminating generation/scoring.
Illustrative example: Prepare teacher scores per prefix and reread them for multiple updates.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why isn't cached teacher scoring equivalent to continuously online distillation?
Show answer and example
Answer: Initial student samples are scored under one teacher config. Later updates shift visitation without refreshing labels. Coverage and teacher consistency govern the cost tradeoff; local evidence is data flow only.
Illustrative example: Cached early error prefixes help initially but later new approaches need an explicit refresh protocol.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does offline cache coverage differ from refreshing rollouts each step?
Show answer and example
Answer: Cache covers initial student visits; updated policy may reach new states. Refreshing covers those at higher cost. Cached isn't always on-policy.
Illustrative example: A newly learned solution's prefixes won't appear automatically in an old cache.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
What Lightning cache identities must match?
Show answer and example
Answer: Teacher checkpoint, prompt, tokenizer, response and position mask. Changes invalidate labels. Local data flow is not Qwen vocabulary distillation or AIME.
Illustrative example: A tokenizer update can misalign stored token scores despite the same filename.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why does cross-script knowledge research test more than translation?
Large Reasoning Models Struggle to Transfer Parametric Knowledge Across Scripts
Show answer and example
Answer: Another script can affect entity recognition, retrieval of parametric facts and reasoning expression. Translation is one factor. Regression/interventions separate them; this isn't a new RL algorithm.
Illustrative example: Failing on a transliterated name doesn't prove the model never learned facts about that person.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why doesn't mechanical transliteration determine whether knowledge exists?
Large Reasoning Models Struggle to Transfer Parametric Knowledge Across Scripts
Show answer and example
Answer: Script changes can affect identity recognition, retrieval and expression, which before/after outputs cannot disentangle. The paper uses regression, training entity prompts and actual LoRA. Local reference notes are not capability.
Illustrative example: Failure on a name in another script may be entity mismatch; hints diagnose but don't establish transfer training.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
Can inference transliteration replace training-time entity interventions?
Large Reasoning Models Struggle to Transfer Parametric Knowledge Across Scripts
Show answer and example
Answer: No. Training interventions study retrieval/learning pathways; transliteration changes current input and may bypass difficulty. String rules or a small classifier cannot stand in for LoRA and semantic transfer.
Illustrative example: Normalizing names to a familiar script can help without teaching cross-script transfer.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Is there an executable local knowledge transfer operator?
Large Reasoning Models Struggle to Transfer Parametric Knowledge Across Scripts
Show answer and example
Answer: No. Literature only; real LM semantics and LoRA training are not implemented. Equations/generic SFT don't imply reproduction, and prompts must not include test answers.
Illustrative example: A working transliteration function is not the paper's parametric retrieval mechanism.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Where does PLD store knowledge without parameter fine-tuning?
Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning
Show answer and example
Answer: The teacher abstracts, clusters and revises readable system rules. A student uses frozen instructions at inference. Rules are learned artifacts despite unchanged weights.
Illustrative example: Package date-processing lessons as system instructions.
Sections in the Chinese source: 背景与主要改动, 核心公式与数据协议. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why separate train/validation/test without student weight updates?
Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning
Show answer and example
Answer: Labeled examples shape rules through extraction, clustering and revision. They carry learned information. Validate and freeze, then never patch from test answers. PLD compiles reasoning into instructions, not parameters.
Illustrative example: Freeze date normalization after validation; adding a test-gold exception leaks even without gradients.
Sections in the Chinese source: 背景与主要改动, 核心公式与数据协议. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does semantic DBSCAN differ from string deduplication?
Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning
Show answer and example
Answer: Cluster paraphrased micro-rules and filter noise rather than only exact repeats. Conflicts still require training-error revisions; lists don't ensure consistency.
Illustrative example: Differently worded unit-conversion rules can share a cluster.
Sections in the Chinese source: 背景与主要改动, 核心公式与数据协议. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
When freeze PLD rules, and can test feedback revise them?
Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning
Show answer and example
Answer: Extract/repair on train, use validation for convergence and freeze before test. Prompt changes can leak without weight updates; track versions/provenance.
Illustrative example: Adding a test-specific date exception and rescoring is not independent evaluation.
Sections in the Chinese source: 背景与主要改动, 核心公式与数据协议. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why anneal CHORD SFT auxiliary weight during RL?
Show answer and example
Answer: Expert demonstrations stabilize early exploration; excessive later imitation can restrict it. Dynamic weighting and token uncertainty ease the handoff without abrupt shift.
Illustrative example: Begin with more standard examples, then let outcome reward guide increasingly.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why taper expert supervision instead of switching off SFT at once?
Show answer and example
Answer: Early guidance stabilizes exploration, but prolonged imitation constrains it. Annealing and uncertainty weights allow a gradual transition while monitoring source shift. Blending isn't universally better; local proxy isn't an uncertainty model.
Illustrative example: Give learner less expert help as it improves rather than always equal help or an abrupt cutoff.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
Is CHORD the same as PIVOT's per-question switching?
Show answer and example
Answer: No. CHORD blends losses with annealed supervision; PIVOT routes prompts by teacher perplexity to separate objectives. Granularity and schedule differ.
Illustrative example: One can apply both losses to a prompt; the other chooses its route.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Does local CHORD gold candidate model real token uncertainty?
Show answer and example
Answer: No. Deterministic expert proxy; no LLM uncertainty model. Track expert off-policy vs student rollout and anneal as specified; keep expert answers out of evaluation inputs.
Illustrative example: Fixed candidate labels don't estimate uncertainty at every long-text position.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why let one SPIRAL model propose and solve games?
Show answer and example
Answer: An adversarial, decidable zero-sum game creates a curriculum as opponents evolve. Roles need distinct rewards and explicit rules; proposer cannot declare outcomes.
Illustrative example: One role creates a position, another tries to solve it, and a program judges.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why require automatically decidable game rules in zero-sum SPIRAL?
Show answer and example
Answer: Adversarial progress creates curriculum, but independent rules must decide wins. Zero-sum incentives do not guarantee broad reasoning coverage. Local state/objective checks don't reproduce self-play training.
Illustrative example: The proposer can't change rules midgame to make every solution lose.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
What are limits of self-play vs fixed hard-problem banks?
Show answer and example
Answer: Self-play adapts but can get trapped in game-specific tricks. Fixed banks have defined coverage but no evolving opponent. Game gains don't imply general reasoning.
Illustrative example: Winning a word game needn't improve math, retrieval or tools.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Why preserve zero-sum rewards across SPIRAL roles?
Show answer and example
Answer: One outcome gives opposite credit under aligned states/rules. Rewarding both for the same victory changes the game. Local kernel isn't large multi-round training.
Illustrative example: A proposer win cannot also be a solver win.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why does Absolute Zero still need an executable verifier?
Show answer and example
Answer: Generated tasks and solutions aren't truth. Program execution provides independent reward. No human item labels isn't no supervision; task executability and verifier quality bound scope.
Illustrative example: Run generated code rather than trust its claimed output.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
Why does a self-generated Absolute Zero curriculum still need verification?
Show answer and example
Answer: Proposer creates tasks, solver answers and program execution supplies independent rewards, replacing human annotations. Verifiable tasks still supervise; exposing saved answers breaks the loop. Local diagnostics aren't full training.
Illustrative example: Generate executable code and predict output, then run it to grade instead of showing stored stdout.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does making harder tasks differ from a capability-boundary curriculum?
Show answer and example
Answer: Too hard can yield all failures and no signal. Use verifiable challenges near current ability. Legal format and novelty don't establish teaching value.
Illustrative example: A task no one can solve at the budget may be worse for learning than a moderate challenge.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
Can Absolute Zero show solver answers saved during task generation?
Show answer and example
Answer: No. Keep execution results on verifier side; solver sees public task only. Local curriculum/state checks aren't large-model capability evidence.
Illustrative example: Don't append actual output to the program-output prediction prompt.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Basics
Why keep generating fresh RAFT candidates from current policy?
Show answer and example
Answer: Fixed SFT data can't follow improvement. Resample, rank by reward and supervise better behavior next round. Coverage still limits which outputs can be chosen.
Illustrative example: Improved summarization can generate better candidates for later training.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Core mechanism
How does reward selection followed by maximum likelihood differ from direct policy gradients?
Show answer and example
Answer: Rank sampled replies with the reward model and supervise selected text; scores determine selection, not per-action gradient weights. Resample with the next policy. Judge bias and missing candidates persist; local tests cover the iteration state.
Illustrative example: Learn the highest-rated candidate summary; absent good summaries can't be chosen and judge errors can be copied.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Method comparison
How does reward enter RAFT differently from PPO?
Show answer and example
Answer: RAFT uses reward to select texts and then applies maximum likelihood. PPO uses advantages and probability ratios directly in policy objective. Selection and continuous weighting differ.
Illustrative example: A high score can decide archive admission without proportionally increasing CE.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Data and budgets: where to spend the next unit of compute
Implementation details
How isolate RAFT selection from evaluation?
Show answer and example
Answer: Log reward model and rollout version, select among same-prompt candidates then update policy. No test answers for sorting. Local candidates aren't a real generator or reward model.
Illustrative example: Training on choices made with test correctness invalidates held-out gains.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Topic guide and reading order
Start with this question¶
Should a limited budget buy more questions, more rollouts, or more supervision at valuable positions?
Takeaway¶
Question selection changes visited states, curricula change difficulty, and trimming supervision or rollout reduces per-example cost. Difficulty estimates age as the policy improves. Separate data selection from target tasks and a fixed evaluation set. LESSER makes selection cheaper through output-layer gradient features; it does not replace the selection objective. Compare quality under the total budget and identify which step actually gets cheaper.
Example¶
With 100 answers per day, more rollouts on solved questions may add little; unsolvable questions may provide no positive rewards. Re-estimate success on training data, sample the difficulty band with remaining signal, and retain a hard control. If rollouts end halfway, token inclusion probabilities change and the old unweighted objective may no longer apply.
Common pitfall¶
Faster is not better learning and fewer tokens are not matched quality. Include teacher forward passes, question synthesis, screening, and retries in cost. Select curricula or budgets on validation and report isolated test. CPU gradient and clipping checks do not prove training speedup, and cross-script transfer research is not an executable optimizer.
Reading order¶
Read GRPO Difficulty Curriculum for policy-dependent difficulty, LESSER for cheap gradient features versus selection rules, and STOPD for early stopping with explicit sampling and weighting.
- A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPO · Original paper
- LESSER: Post-Training Data Selection with Output-Layer Gradients · Original paper
- Does On-Policy Distillation Need to Complete Every Rollout? · Original paper
Check your understanding¶
Can we keep the previous round's “medium difficulty” label after the policy changes?
Show suggested answer
Not by default. Difficulty depends on the current policy and sampling budget. Re-estimate under the chosen curriculum using permitted training/validation information, not final test. If budget forces reuse, state that limitation; fixed prompt length is not success-probability difficulty.