Policy optimization: which policy samples and which policy learns?¶
简体中文 · English
LLM post-training / 220 questions · 55 papers
Try answering each question before opening the explanation. The bank grows as papers and questions are added.
Read this topic's comparisons, analysis, and cross-paper Q&A
220 questions · 55 papers
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
How does a RLiD action differ fundamentally from an LLM writing an action command?
RLiD: Reinforcement Learning in the Decision Plane for Language Agents and LLMs
Show answer and example
Answer: RLiD normalizes over compiled executable single-token menu labels and selects an action in one forward pass, without generating or parsing free text. Benefits depend on the menu and observation interface. It does not automatically preserve the extra computation provided by long reasoning, or guarantee stronger performance on every task.
Illustrative example: Teaching example: With only open and take in the menu, label B maps directly to take; free text might contain only an explanation. Legal output follows from an interface contract, not 100% task success.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Does RLiD's variance theorem prove a newly trained LDM must beat a generative LLM?
RLiD: Reinforcement Learning in the Decision Plane for Language Agents and LLMs
Show answer and example
Answer: No. Within one text policy, the identity marginalizes alternative wordings that map to the same action. Given that action, advantage must not depend on wording. It preserves expected updates and removes realization variance, not differences between model parameterizations or training runs, nor temporal-credit noise.
Illustrative example: Teaching example: open drawer and I should open drawer can be marginalized into one action. If reward separately values polite wording, advantage depends on wording and the assumption fails.
Sections in the Chinese source: 核心公式, 数据协议与代码映射. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Is KL from uniform to the menu marginal equivalent to making every decision more random?
RLiD: Reinforcement Learning in the Decision Plane for Language Agents and LLMs
Show answer and example
Answer: No. It first averages microbatch action distributions within a menu group, then encourages that label marginal toward uniformity. Individual decisions can remain confident. Per-decision entropy instead encourages each row to spread out, changing the target. External calibration subtraction is another label-bias treatment with a separate update schedule.
Illustrative example: Teaching example: Decisions near [1,0] and [0,1] average to [.5,.5], giving zero marginal KL despite low entropy in each row.
Sections in the Chinese source: 核心公式, 结果、负控与边界. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Do executable menus and clipped PPO complete RLiD agent training?
RLiD: Reinforcement Learning in the Decision Plane for Language Agents and LLMs
Show answer and example
Answer: No. Old-policy log-probabilities and advantages must come from actual external sampling, stay frozen, and align with the same menu. Local code supplies no LM, compiler, memory, GiGPO, or distillation. With mean-episode-length divided by each episode's length as weights, average weighted decisions without arbitrarily dividing again by total weight.
Illustrative example: Teaching example: A two-step episode with .75 per step and a one-step episode with weight 1.5 both total 1.5. This checks weighting, not the paper's 99.3% ALFWorld full-route result.
Sections in the Chinese source: 数据协议与代码映射, 结果、负控与边界, 高成本完整实验. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why progressively shorten MOER expert prefixes?
MOER: Mix-Policy Optimization under Expert Prefix Guidance with Selective Rollback
Show answer and example
Answer: Complete solutions limit exploration, but no guidance can be too hard. Shortening transfers more continuation to the student. If continuation fails, roll back to a verified complete expert trace rather than treating failure as demonstration.
Illustrative example: Give two steps, then one, leaving more of the derivation to the student.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why is a student continuation after an expert prefix still off-policy?
MOER: Mix-Policy Optimization under Expert Prefix Guidance with Selective Rollback
Show answer and example
Answer: It is conditional on a supplied prefix, not a complete sample from unguided student. MOER keeps successful guided traces and falls back to verified expert traces on failure, using distinct objectives. Expert data stay in training.
Illustrative example: Completing a proof after the teacher provides the key step is not reaching that state independently.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does MOER distinguish native on-policy and guided trajectories?
MOER: Mix-Policy Optimization under Expert Prefix Guidance with Selective Rollback
Show answer and example
Answer: The whole guided trace is off-policy even if the student wrote the suffix, because its starting state was externally influenced. Use different objectives, not token authorship to reclassify the suffix.
Illustrative example: A student writing step three after two expert steps remains in the guided trace.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
How can MOER's off-policy ratio equal one forward yet have gradients?
MOER: Mix-Policy Optimization under Expert Prefix Guidance with Selective Rollback
Show answer and example
Answer: The denominator is detached and numerator differentiable, so equal values do not cancel derivatives. Joint group advantages are not divided by SD; normalize over all valid response tokens, not extra mixture-weighted counts.
Illustrative example: A numerator divided by its detached copy is one, but differentiating numerator still changes loss.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why does CORD select a protected objective first?
CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents
Show answer and example
Answer: Adding conflicting task and teacher gradients can cancel them. The protected direction states whose first-order progress must not be harmed; modify the other minimally. It is a tradeoff.
Illustrative example: Protecting task reward versus teacher imitation defines different projections.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why project one gradient relative to a chosen protected gradient?
CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents
Show answer and example
Answer: Minimally change the conflicting other vector to preserve the selected objective. Swapping protection changes updates. Conflict-only and always differ; project after full-batch accumulation. This is pre-optimizer first-order geometry, not guaranteed finite AdamW descent.
Illustrative example: Task reward and imitation may push a parameter oppositely; changing which is protected changes what survives.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How do CORD conflict-only and always-project variants differ?
CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents
Show answer and example
Answer: Conflict gating keeps helpful aligned components and projects only for negative dot product. Always projecting removes even helpful parallel components. The resulting direction is different; no finite adaptive-optimizer descent guarantee.
Illustrative example: Aligned gradients add directly under conflict gating; always-project still removes a parallel component.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Why aggregate a full CORD optimizer batch before projecting once?
CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents
Show answer and example
Answer: Projection is nonlinear, so per-microbatch projection and summation usually differ. After fixed aggregation apply objective weight λ first. Skip projection if protected gradient is zero; λ=0 returns first objective direction.
Illustrative example: Conflicts in two microbatches can cancel in the aggregate; early projection changes the result.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why train ORL diffusion on its own induced residual contexts?
On-Policy Residual Learning for Diffusion Language Models
Show answer and example
Answer: The previous prediction creates the next residual and carries model errors at inference. Ideal residuals induce a state mismatch; online collection and replay expose encountered states.
Illustrative example: A denoising step must see wrong prior probabilities to learn recovery from them.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why replay model-generated residual states instead of fixed residual inputs?
On-Policy Residual Learning for Diffusion Language Models
Show answer and example
Answer: Predictions become subsequent context and propagate errors. Fixed/ideal training can miss the model's inference-state distribution. Online collection and exact replay with entropy contraction address this during training, not through final-output fixes. No local full training evidence.
Illustrative example: An early wrong token bias affects the next denoising step; perfect residuals may never teach this correction.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does ORL differ from smoothing outputs after inference?
On-Policy Residual Learning for Diffusion Language Models
Show answer and example
Answer: Postprocessing changes final outputs only. ORL feeds previous distributions into later model states and trains on them, including exact replay and entropy contraction; simple averaging is not equivalent.
Illustrative example: Averaging two answers doesn't teach denoising from a mistaken previous state.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Can the repository call an ORL adapter for residual traces?
On-Policy Residual Learning for Diffusion Language Models
Show answer and example
Answer: No. Literature-only: no adapter, metrics or GPU receipt. Equations are explanatory; generic diffusion or a registry name does not implement online collection and exact replay.
Illustrative example: Adding an evolve key cannot create the trajectory process.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why distinguish GGPO behavior and reference policies?
Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards
Show answer and example
Answer: Behavior generates the sampled data; reference anchors KL. They needn't match. Refreshing reference changes the later objective without resampling old data.
Illustrative example: One frozen policy collects replies and another anchors deviation penalties.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why can't GGPO behavior and reference be treated as the same role?
Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards
Show answer and example
Answer: Behavior is the sample source; reference defines KL-anchored Gibbs objective. Optimize adjusted log-mean-exp vs mean without extra behavior IS. Reference refresh cannot rewrite sampling provenance; filtered/normalized logs are not raw theorem quantities.
Illustrative example: A batch collected by an old policy remains old-policy data after changing the optimization anchor.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does GGPO's Gibbs gap differ from learner/behavior importance weighting?
Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards
Show answer and example
Answer: Differentiate adjusted-reward log-mean-exp minus mean within groups. No extra learner/behavior ratio multiplies updates. Off-policy data alone do not justify adding one.
Illustrative example: Old-policy data do not imply every objective must use the same importance ratio.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Why not backpropagate directly through clipped GGPO LME?
Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards
Show answer and example
Answer: Compute frozen coefficients with clipped learner/reference ratios, then update original sequence logp. Backpropagating through clamp zeroes saturated gradients and changes the surrogate. Keep full-batch normalization/filtering.
Illustrative example: Clipping weight construction doesn't remove saturated positions from learning.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why track prefix-state shift rather than only the current ratio in trajectory TAPO?
Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL
Show answer and example
Answer: Small earlier shifts accumulate into later visitation changes. A current action ratio misses prior branching. Prefix geometric log-ratios supply trajectory context with tighter position-dependent bounds.
Illustrative example: A similar final-action ratio does not mean earlier branches left its state unchanged.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why stop gradient through TAPO's prefix geometric ratio?
Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL
Show answer and example
Answer: The ratio measures state drift, but current credit should not backpropagate into all earlier tokens. Freeze the geometric weight and preserve the current log-probability gradient separately. This avoids cross-position gradients and length dilution.
Illustrative example: Adjust step ten for prefix drift without rewriting credit for steps one through nine.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does prefix geometric mean differ from multiplying probability ratios?
Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL
Show answer and example
Answer: Exponentiate average log ratio to limit extreme length scaling. This is a different estimator, not an equivalent numerically stable product.
Illustrative example: Many small per-step shifts can create an extreme product while the geometric mean retains typical scale.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
How does TAPO prevent prefix weighting from dividing current-token gradients by position?
Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL
Show answer and example
Answer: Detach the geometric ratio and multiply by current probability divided by its detached copy. Forward keeps the weight; backward follows only current action, without prior gradients or inverse-prefix dilution.
Illustrative example: Step ten can use ten-step context without automatically receiving one-tenth gradient.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
What two independent choices define GPPO's proximal updates?
Show answer and example
Answer: Choose f-divergence geometry and its estimation topology. Combinations allow different ratio ranges; a common framework does not make instances train identically.
Illustrative example: Different treatment of complement probability can change trust regions for the same divergence.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why can the same f-divergence give different trust regions with MC, atomic or binary estimation?
Show answer and example
Answer: Topology decides how action and complement probabilities enter. Even fixed function/radius need not yield the same ratio interval. GPPO separates these choices and uses the resulting bounds. Preserve different KL forms instead of renaming GRPO.
Illustrative example: Constraining only a rare action versus also the remaining mass can allow different changes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Why are MC, atomic and binary topologies more than implementation details?
Show answer and example
Answer: They count sampled actions and unsampled complements differently, changing estimated divergence and admissible ratios. Name mappings establish no equivalence without the matching definitions.
Illustrative example: Binary accounts for one minus action probability, not just a renamed atomic result.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Why expose literal KL and two-sided log separately in GPPO?
Show answer and example
Answer: Some algorithm mappings and printed formulas differ; keep both explicit rather than silently fix one. Ratios must be positive and binary complements valid; solve the selected trust region instead of copying PPO bounds.
Illustrative example: A configuration called KL that actually computes two-sided log obscures what was verified.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why put KL drift into ZCPO group coefficients rather than an independent loss?
When KL Regularization Misfires in Group Policy Optimization
Show answer and example
Answer: A standalone KL can update policy with no reward distinction. Center relative KL and rewards into zero-sum coefficients for the existing surrogate and relative group credit.
Illustrative example: Equal rewards should not trigger a training step merely because a separate KL exists.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why use zero-sum KL coefficients instead of adding KL loss?
When KL Regularization Misfires in Group Policy Optimization
Show answer and example
Answer: Independent KL can update when reward gradients cancel or rewards tie. Center average conditional-KL drift with group rewards and feed the existing surrogate. Skip zero-scale/ineligible groups, including optimizer side effects, rather than returning zero loss and stepping.
Illustrative example: All rewards equal gives no relative signal; verify no optimizer update rather than rely on the logged loss.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Can a sampled-token log ratio substitute for ZCPO conditional KL?
When KL Regularization Misfires in Group Policy Optimization
Show answer and example
Answer: No. Use full-vocabulary conditional KL, averaged along each response and centered across group. A sampled ratio is a different noisy estimator.
Illustrative example: A rare sampled token's drift does not describe the entire conditional distribution.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Why skip optimizer updates for all-equal rewards in ZCPO?
When KL Regularization Misfires in Group Policy Optimization
Show answer and example
Answer: Reward scale is zero, and AdamW decay may move parameters even with a zero loss. Skip the group/step; undersized groups also cannot form relative coefficients.
Illustrative example: Zero logged gradient plus weight decay can still alter the model.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why measure both first-attempt success and fixed-budget coverage in Sharpening Tax?
Sharpening Tax in Post-Training
Show answer and example
Answer: Training may raise probability of familiar successes while making other solutions or tasks unreachable. Fixed-budget coverage exposes this tradeoff.
Illustrative example: One problem becomes usually correct while another never yields a correct sample, reducing coverage.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why measure fixed-budget task coverage even when pass@1 improves?
Sharpening Tax in Post-Training
Show answer and example
Answer: Sharper distributions hit familiar solutions more often but may lose rare solutions accessible with several draws. Compare base/post coverage under equal budget and adapt temperature by difficulty. First-hit accuracy, entropy and task coverage are not interchangeable.
Illustrative example: Better common-item answers can hide lost rare correct solutions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does posterior-conditioned temperature differ from uniformly raising temperature?
Sharpening Tax in Post-Training
Show answer and example
Answer: Allocate temperature by estimated difficulty to balance exploitation and coverage. Global heating treats all tasks alike. Bad estimates may misallocate, so recovery isn't guaranteed.
Illustrative example: Mastered problems may need little exploration while difficult problems benefit from branching.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Why hold K and task inputs fixed when computing Tax_K?
Sharpening Tax in Post-Training
Show answer and example
Answer: Coverage changes with samples and tasks. Match both and retain the sign of base minus post-training coverage. Local code is a metric only.
Illustrative example: Many base samples against one post-training sample confound the estimated tax.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why not apply a uniform length penalty to all long replies in RLCPR?
RLCPR: Rethinking Probability-Based Reinforcement Learning From Posterior Concentration
Show answer and example
Answer: Long reasoning can be necessary. Enable relative penalty only when all group replies are long and answer posteriors concentrated, where length may add little distinction.
Illustrative example: Repeated explanation after agreement differs from exploring uncertain critical branches.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why penalize relative length only if all answers are long and posteriors concentrated?
RLCPR: Rethinking Probability-Based Reinforcement Learning From Posterior Concentration
Show answer and example
Answer: Similar posteriors indicate limited distinction from reasoning length; widespread long replies support compressing redundant ones. Penalize relatively only after both gates. Equality and quantile rules matter; real entropy/posteriors must come from a model.
Illustrative example: Favor shorter among equally supported long replies, not a long reply that clearly increases answer support.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Are RLCPR entropy strata and length gating the same choice?
RLCPR: Rethinking Probability-Based Reinforcement Learning From Posterior Concentration
Show answer and example
Answer: Entropy strata shape examples before sampling. Length/posterior gates adjust rewards inside generated groups. Different inputs and stages, not one global length threshold.
Illustrative example: Deciding which tasks to sample precedes checking whether current replies are all redundant.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Do exact length or posterior thresholds activate RLCPR?
RLCPR: Rethinking Probability-Based Reinforcement Learning From Posterior Concentration
Show answer and example
Answer: No. Minimum length must strictly exceed its threshold and posterior range must be strictly below its bound. Entropy CDF also uses strict less-than on ties. Handcrafted numbers test formulas, not RL gains.
Illustrative example: Minimum length exactly at threshold is not penalized.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why use reward intervals instead of point scores in Range-GRPO?
Range-GRPO: Policy Optimization via Pairwise Relations among Reward Intervals
Show answer and example
Answer: A point hides judge uncertainty. Intervals identify pair orderings that can be justified and avoid treating tiny noise as preference.
Illustrative example: Strongly overlapping score ranges need not produce a winner.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why not rank replies with overlapping reward intervals?
Range-GRPO: Policy Optimization via Pairwise Relations among Reward Intervals
Show answer and example
Answer: Small point differences may be judge noise. Use determinate calibrated interval relations and aggregate relative group advantage without inventing directions. Degenerate intervals recover point rewards. The kernel is not judge calibration.
Illustrative example: Do not favor one broad overlapping interval merely because its center is higher.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does treating overlap as uncertain differ from midpoint ranking?
Range-GRPO: Policy Optimization via Pairwise Relations among Reward Intervals
Show answer and example
Answer: Midpoints always impose order even inside large uncertainty. Interval comparison discards dubious direction, at the cost of sparser supervision. It does not prove faster/better training.
Illustrative example: Different midpoints with overlap needn't imply a clear preference.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Does local Range-GRPO calibrate a conformal judge?
Range-GRPO: Policy Optimization via Pairwise Relations among Reward Intervals
Show answer and example
Answer: No. It consumes intervals and tests comparisons and the point limit. Coverage needs real calibration data/judging. Degenerate intervals should recover the DR-GRPO limit, not arbitrary smoothing.
Illustrative example: A wide interval passing an interface test doesn't establish coverage.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why cap total group importance mass in GMC-GRPO?
Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis
Show answer and example
Answer: Asynchronous stale policies can produce huge ratios. Apply one multiplier to the group to cap mass while preserving within-group proportions, unlike independent clipping.
Illustrative example: Scale the whole inflated group instead of cutting only its largest member.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why rescale an entire group instead of clipping trajectories one by one?
Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis
Show answer and example
Answer: Independent clipping changes relative weights. A shared factor limits total ratio mass and retains relative contributions. Sampling and moment assumptions remain; a formula test is not an asynchronous training system.
Illustrative example: Uniformly scale stale trajectories when total mass is too large; separate caps can flatten differences.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
What does a group cap preserve that per-trajectory clipping may not?
Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis
Show answer and example
Answer: Common scaling retains ratios between weights. Independent caps change them. Theory still requires conditions such as common second moments; it does not repair arbitrary staleness.
Illustrative example: Clipping may make large and medium weights equal, while group scaling preserves their ratio.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
How do GMC weights behave below the total cap?
Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis
Show answer and example
Answer: Use multiplier one until total exceeds cap; then multiply by cap divided by sum. Check nonnegative weights and group boundaries. Local kernel does not measure asynchronous cluster throughput.
Illustrative example: Do not pool two prompts: a large weight in one should not scale the other.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Does RFPO reward-free mean no quality signal?
Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
Show answer and example
Answer: A calibrated frozen critic predicts prefix success instead of costly external terminal rewards. Supervision moves into the critic, whose reliability still matters.
Illustrative example: An unfinished solution can get a success estimate because of prior evaluation learning, not by knowing the answer from nothing.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why debias and binarize a frozen critic instead of maximizing its continuous score?
Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
Show answer and example
Answer: Freezing stops parameter drift, not bias. Policies can exploit length or other shortcuts. Debias and binarize prefix success estimates while using the critic for baselines and unfinished forecasts. Reward-free reuses supervision rather than removes it.
Illustrative example: A length-biased critic is exploitable even when frozen.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
What risk comes from sharing a critic for reward and baseline?
Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
Show answer and example
Answer: Shared bias can be exploited by the policy. Length debiasing and binarization limit shortcuts without ensuring attack resistance. Critic score itself isn't evidence of capability gains.
Illustrative example: If the critic favors verbosity, the policy may simply lengthen responses.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Why freeze the RFPO critic and distinguish unfinished forecasts?
Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
Show answer and example
Answer: Freezing keeps the evaluation baseline stable. A forecast handles missing terminal outcomes, not verified success. Local debiasing, binary reward and GAE kernels do not train a long-reasoning model.
Illustrative example: A high estimate on a truncated trace remains an estimate, not an answer labeled correct.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
When does GRAFT borrow a peer group?
Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR
Show answer and example
Answer: When receiver candidates all fail and a peer group for the same prompt contains both success and failure. Only then is there transferable relative credit; it is not copying one answer from the best model.
Illustrative example: Four receiver failures can use a peer group with both correct and incorrect replies.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why exchange full same-prompt peer groups instead of only a winning answer?
Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR
Show answer and example
Answer: A receiver all-fail group lacks differentiation; the whole peer group preserves successes, failures and their relative credit. Cross-model/tokenizer mismatch needs sequence compatibility and token clipping. Do not mark borrowed data on-policy.
Illustrative example: Give the receiver a peer's complete group and credit rather than inserting a single gold answer.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Why borrow a complete peer group rather than one success?
Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR
Show answer and example
Answer: It retains internal advantages. Adding only a positive changes the group distribution and credit baseline. Borrowed trajectories are not receiver on-policy data.
Illustrative example: Keep peer failures to preserve why peer successes have relative positive advantage.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Why use average sequence likelihood for cross-tokenizer GRAFT compatibility?
Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR
Show answer and example
Answer: Token counts and boundaries differ, so use sequence-level compatibility then ratio clipping to bound updates. Track text rescoring, provenance and group advantages. This doesn't make policies identical.
Illustrative example: The same sentence tokenized differently cannot be compared by subtracting probabilities position by position.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why model turns rather than average all tokens in TLM-DRE?
Turn-level Multiscale Density Ratio Estimation for LLM Agents
Show answer and example
Answer: Turns have different roles and lengths. Multiscale weighting preserves dialogue structure; token averaging may let long turns dominate. Actual credit depends on density ratios and feedback.
Illustrative example: A short crucial tool choice should not be drowned out by a verbose explanation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why upweight low-confidence TLM-DRE turns?
Turn-level Multiscale Density Ratio Estimation for LLM Agents
Show answer and example
Answer: Length-normalized reference logp proxies how well a turn is learned. Lower confidence receives greater scale, changing multi-turn density ratios and asymmetric pressure. Low likelihood does not mean wrong. Local diagnostics do not establish generalization.
Illustrative example: Correct a hesitant tool decision more strongly than a confident recap, while recognizing rare valid actions can be unlikely.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does asymmetric positive/negative density weighting differ from uniform weights?
Turn-level Multiscale Density Ratio Estimation for LLM Agents
Show answer and example
Answer: Positive and negative evidence reinforce and suppress differently. A common curve removes that design, but asymmetry still needs real multi-turn evaluation.
Illustrative example: Rewarding a correct action and suppressing a wrong continuation need not share weights.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Does the local TLM-DRE kernel demonstrate real multi-turn agent training?
Turn-level Multiscale Density Ratio Estimation for LLM Agents
Show answer and example
Answer: No. It computes reference turn weights and asymmetric token updates without an agent environment. Retain turn memberships; concatenating into one turn changes the method.
Illustrative example: Three tool turns collapsed into a string cannot test the original allocation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why target output classes rather than token entropy in MaD-RL?
MaD-RL: Matching Distributions for Calibrating LLMs with Reinforcement Learning
Show answer and example
Answer: Classes measure semantic result frequencies; token entropy measures surface uncertainty. Many wordings can yield one class. Combine empirical-to-target class mismatch with correctness.
Illustrative example: Many phrasings recommending the same item category do not improve category coverage.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why match output-class frequencies instead of just raising token entropy?
MaD-RL: Matching Distributions for Calibrating LLMs with Reinforcement Learning
Show answer and example
Answer: Randomized words can collapse to the same decision. Map outputs to classes and reward empirical-to-target similarity plus correctness. Hold frequencies fixed per update; unsampled classes get no fictitious on-policy examples. Specify invalid-output treatment.
Illustrative example: Check decisions directly even when language appears diverse.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
What tradeoff does distribution matching add beyond maximizing accuracy?
MaD-RL: Matching Distributions for Calibrating LLMs with Reinforcement Learning
Show answer and example
Answer: Accuracy can favor easy classes, while matching moves mass toward target frequencies. Report both: matching doesn't guarantee correctness and high accuracy doesn't guarantee target ratios.
Illustrative example: Balanced classes can still include wrong replies in one category.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Can invalid outputs be dropped from MaD-RL frequency denominators?
MaD-RL: Matching Distributions for Calibrating LLMs with Reinforcement Learning
Show answer and example
Answer: Not arbitrarily; doing so changes empirical frequencies and rewards. Treat frequencies as fixed per update, smooth zero target probabilities explicitly, and do not invent gradients for unsampled categories. Local diagnostics aren't full training.
Illustrative example: Dropping all unclassifiable samples can make remaining frequencies appear closer to target.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Does TASCO adapt backbone weights or a lightweight prefix?
Beyond Confidence: Stability-Aware Test-Time Adaptation for LLM Reasoning
Show answer and example
Answer: Freeze the backbone and optimize an added prefix, changing conditioning without retraining the model or selecting only by sampling. It uses unlabeled current-task signals, never test gold.
Illustrative example: Update prefix vectors and reread probabilities while leaving model layers and answer labels untouched.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why check stability in addition to confidence during TASCO adaptation?
Beyond Confidence: Stability-Aware Test-Time Adaptation for LLM Reasoning
Show answer and example
Answer: Confidence can reinforce wrong answers. Keep backbone fixed, tune a small prefix and penalize unstable predictions under neighborhood perturbations. Stability is unlabeled evidence, not a correctness oracle; no real local LM prefix training.
Illustrative example: If equivalent wording changes the answer, high original confidence alone is a poor signal.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How do TASCO prefix updates and best-of-N spend extra compute differently?
Beyond Confidence: Stability-Aware Test-Time Adaptation for LLM Reasoning
Show answer and example
Answer: TASCO optimizes conditioning before a frozen model predicts; best-of-N generates and selects more candidates. Compare optimizer forwards/backwards and generation, not just candidate count.
Illustrative example: Several gradient steps and several independently generated answers are different workloads.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Does the local TASCO stability penalty train an actual LM prefix?
Beyond Confidence: Stability-Aware Test-Time Adaptation for LLM Reasoning
Show answer and example
Answer: No. It implements random neighborhood perturbation and a stability reference kernel without a frozen LM prefix path. Small-vector tests do not establish test-time gains.
Illustrative example: Stable numeric features do not show a language model's prefix was optimized on tasks.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why does Probe-ERPO use probes independent of final answers?
Show answer and example
Answer: Gold is unavailable during test-time adaptation. Using it would defeat unlabeled adaptation. PCR uses probe consistency to decide reinforcement vs negative constraints.
Illustrative example: Check agreement from intermediate states instead of matching to gold.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why use answer-independent probe consistency and not reinforce the majority at low agreement?
Show answer and example
Answer: Self-labeling can repeat errors. Output-free probes estimate consistency; high/low branches differ. Low agreement constrains lower ranks while entropy preserves exploration. Consistency is not supervised correctness.
Illustrative example: Conflicting probes justify suppressing weak options without blindly rewarding the highest vote count.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Why aren't high-PCR reinforcement and low-PCR negative ranking ordinary pseudo-labeling?
Show answer and example
Answer: Low consistency supplies no reliable positive, so constrain poor candidates rather than canonize the majority. High agreement is only a clue; entropy control and independent evaluation still matter.
Illustrative example: When candidates conflict, avoid weak ones without declaring one correct.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
May Probe-ERPO's negative rank mask read the answer key?
Show answer and example
Answer: No. Use allowed probes and candidate rankings only; gold appears in independent scoring. A fixed mask validates the update path, not real probe discrimination.
Illustrative example: A fixture can specify which positions to suppress without teaching automatic error detection.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why train TTPO majority-agreeing and dissenting trajectories separately?
TTPO: Test-Time Policy Optimization
Show answer and example
Answer: The majority may be wrong, and dissent provides a different signal. Use self-distillation for agreement and grouped RL for disagreement rather than one unlabeled pseudo-target.
Illustrative example: A majority answer and an alternative path do not share the same imitation objective.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why use different objectives for majority agreement and disagreement?
TTPO: Test-Time Policy Optimization
Show answer and example
Answer: Copying consensus can spread errors. Self-distill agreeing replies, apply group RL to dissent, and filter already converged positions and high-confidence errors. No gold or claim that votes are truth.
Illustrative example: Don't delete the dissenting answer outright; handle it under the relevant objective and reliability filters.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
What safeguards does TTPO add instead of treating the majority answer as gold?
TTPO: Test-Time Policy Optimization
Show answer and example
Answer: Asymmetric objectives filter converged tokens and confident errors. It recognizes that consensus is not truth; filter quality matters and consensus isn't independent validation.
Illustrative example: A group can confidently agree on the same wrong answer.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Does local TTPO perform real test-time sampling and Qwen updates?
TTPO: Test-Time Policy Optimization
Show answer and example
Answer: No. It tests branch objectives and token selection only. Fixed candidates are not real unlabeled adaptation. Keep gold hidden before branch assignment.
Illustrative example: Supplying answers for formula checks does not mean the model generated them on benchmark prompts.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
What resources do Merge, Mix RL and MOPD reuse in RLVR Fusion?
Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms
Show answer and example
Answer: Merge reuses expert weight deltas; Mix RL reuses training data; MOPD distills from experts using data. Resource requirements differ, so averages cannot choose a route alone.
Illustrative example: Without original training data, available routes differ even if expert weights are present.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why are Merge, Mix RL and MOPD different fusion operations?
Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms
Show answer and example
Answer: Merge combines expert parameter increments, Mix RL trains on combined data, and MOPD uses expert guidance with data. Availability and budget determine feasible routes. Similar means or a probability-vector proxy do not establish lossless checkpoint fusion.
Illustrative example: Experts alone permit discussing parameter merging; data without experts cannot run the same distillation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Why doesn't similar average performance imply per-domain equivalence?
Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms
Show answer and example
Answer: Averages can cancel larger task gaps. Compare target domains, available experts/data and compute, not only mean scores or similar formulas.
Illustrative example: Better math and worse code can yield the same mean without meeting the same needs.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Why define Merge task vectors relative to a common base?
Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms
Show answer and example
Answer: Expert minus base must share parameter coordinates and reference. Different bases/layouts don't express comparable deltas. Local candidate merging isn't checkpoint merging.
Illustrative example: Experts with different vocabularies or tensor layouts cannot simply average deltas.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why inject a golden response only into GRIN failure groups?
From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection
Show answer and example
Answer: All-fail groups lack positive contrast. Verified solutions supply direction; preserve more native exploration when the student succeeds. Gold is training supervision, not evaluation input.
Illustrative example: Add a verified answer only after all attempts fail, not to every prompt.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why correct GRIN injection by the mixed distribution?
From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection
Show answer and example
Answer: An expert addition was not an unguided current-policy sample. Correct with the mixture denominator; reduce injection as capability improves and return to exploration. Gold must not appear in evaluation/deployment.
Illustrative example: Learn from an injected verified solution without logging it as a student exploration success.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does mixture correction differ from treating the expert answer as a student sample?
From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection
Show answer and example
Answer: Sampling includes the injection source, so the denominator must represent the mixture. Calling it pure on-policy misweights it and hides external knowledge.
Illustrative example: An inserted expert solution was not drawn solely from the student.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Why use q_mix rather than frozen reference in GRIN's importance ratio?
From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection
Show answer and example
Answer: q_mix describes actual sampling; reference anchors regularization. Track mixture origins/probabilities. The deterministic diagnostic does not reproduce the paper's knowledge-acquisition data.
Illustrative example: An old reference is not interchangeable with the expert that generated a sample.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why observe drift on the input side in ERPO?
Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization
Show answer and example
Answer: Response regularization directly limits exploration. ERPO controls induced question-distribution shift with question likelihood weights and Query-KL, moving where regularization applies.
Illustrative example: Balance which questions receive learning pressure rather than forcing old answer wording.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
How does question-side ERPO regularization differ from constraining replies?
Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization
Show answer and example
Answer: Answer KL limits output drift and exploration directly. ERPO weighs queries with reference likelihood and constrains induced question drift, changing update allocation across inputs. It isn't a renamed answer KL; typicality does not verify real query modeling.
Illustrative example: Reweight familiar questions that dominate training rather than constrain every answer to mimic the old model.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Can Query-KL and Policy-KL be interchanged in ERPO?
Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization
Show answer and example
Answer: No. One constrains question distribution, the other conditional replies, with different gradient boundaries. Applying pressure to the answer scoring function isn't input-only control.
Illustrative example: Which question is sampled and how it is answered are different distributions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Why is local ERPO typicality only a query-likelihood proxy?
Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization
Show answer and example
Answer: Finite candidate features approximate input typicality to test flow and gradients, not natural-question LM likelihood. Use frozen reference weights, not weights inferred from test correctness.
Illustrative example: Typical feature frequency isn't the probability of a complete Chinese question.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why protect dominant pretrained singular spaces in GCPO?
GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs
Show answer and example
Answer: Rollout update spikes in principal spaces may precede validation regressions. Restrict updates to bilateral orthogonal complements to avoid overwriting principal mappings. Correlation doesn't forbid learning in all dominant directions.
Illustrative example: Add task behavior in remaining directions while preserving important input/output mappings.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why project low-rank updates onto both singular complements rather than penalize overlap?
GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs
Show answer and example
Answer: Soft cost permits principal-space drift at sufficient reward. Hard bilateral projection excludes it, keeping both input and output directions outside frozen principal spaces. Vector diagnostics are not layered LoRA training.
Illustrative example: Every valid update must avoid protected directions, rather than merely pay a penalty for touching them.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does hard orthogonality differ from a soft projection penalty?
GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs
Show answer and example
Answer: Hard constraints exclude violating updates. Soft penalties allow an optimizer to trade violations against task gains. A larger penalty doesn't establish exact bilateral orthogonality.
Illustrative example: A soft update may briefly enter the principal space; hard projection removes that component every time.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Does local GCPO validate per-layer bilateral LoRA?
GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs
Show answer and example
Answer: No. A vector policy restricts the leading right space of empirical update covariance as a proxy. The paper fixes left and right matrix projections; one-sided vector tests do not verify them.
Illustrative example: A parameter vector lacks two independent matrix axes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why introduce external rollouts in ERILS diffusion RL?
Show answer and example
Answer: Student exploration may lack coverage, while external policies offer different denoising trajectories. Track source and length; high external reward isn't native reachability.
Illustrative example: Log an external model's alternative denoising path as external.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why not normalize native and external rollouts together?
Show answer and example
Answer: Differences in length and rewards can turn provenance into advantage, letting one source dominate. ERILS caps external lengths and treats reward sources separately. This preserves exploration value, not immunity to off-policy errors.
Illustrative example: If outsiders are always longer and higher-scoring, pooled normalization may make every student sample negative.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Why normalize sources separately rather than mix on/off-policy rewards?
Show answer and example
Answer: Sources can differ in reward scale and distribution; pooled baselines can systematically suppress one. Separate normalization preserves within-source credit without solving every off-policy bias.
Illustrative example: A higher expert mean can make useful student improvements negative under one pooled baseline.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Why explicitly enforce external response length in ERILS?
Show answer and example
Answer: Long trajectories consume tokens and change loss scale. Retain length constraints and source labels through normalization. Candidate-policy diagnostics do not reproduce diffusion checkpoint training.
Illustrative example: An oversized expert trajectory could consume the batch and change the intended mixture.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why does ReCo address both duplicate replies and saturated tokens?
Show answer and example
Answer: Common responses dominate sample weight and already-certain tokens can be reinforced repeatedly. Expected response counts and token variance ratios target different redundancies to retain exploration.
Illustrative example: Repeated solutions and their nearly certain format tokens can both dominate gradients.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why downweight common replies and adjust individual-token variance separately?
Show answer and example
Answer: Response weights correct expected repetition across the group; token variance measures local plasticity. Neither replaces the other. Coverage isn't arbitrary randomness; candidate probabilities do not show long-text coverage.
Illustrative example: Downweighting repeated template responses can still leave nearly deterministic final tokens dominating.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Is deduplication equivalent to ReCo expected-count weighting?
Show answer and example
Answer: Hard dedup drops repeated samples; expected-count weights vary continuously with policy likelihood. A separate variance ratio adjusts tokens. The objectives differ.
Illustrative example: Two responses may appear only once each while one has much higher underlying probability and different weight.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Why preserve old versus current probability in ReCo's Bernoulli variance ratio?
Show answer and example
Answer: Compare current p(1−p) to old variance for changes in decision saturation. Protect near-zero denominators, not use current on both sides and get one. Candidate tests aren't text pass@64.
Illustrative example: Check numerical stability when old probability is near a boundary instead of accepting an enormous anomalous signal.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why isn't correctness alone enough for code optimization RL?
Reinforcement Learning for Code Optimization
Show answer and example
Answer: The task also targets speed, but input size and load affect timing. Use dedicated tests and calibration to separate actual improvements from noise.
Illustrative example: Two correct algorithms may show complexity differences only at larger inputs.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why calibrate timing before rewarding speed?
Reinforcement Learning for Code Optimization
Show answer and example
Answer: Noise from workers and small tests can teach exploitation rather than optimization. Independent timing, correctness and performance tests define valid rewards, with sparse/stale handling afterward. Syntax/fixtures don't prove faster production.
Illustrative example: Repeat timing under matched conditions instead of rewarding a run that happened on an idle host.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does runtime reward differ from a math verifier?
Reinforcement Learning for Code Optimization
Show answer and example
Answer: Correctness can be relatively deterministic; time depends on inputs, sandbox and load. Use gates and reference percentiles, not one wall-clock observation.
Illustrative example: A busy shared worker can slow unchanged code without a regression.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Can the local code-optimization component replace a timing service?
Reinforcement Learning for Code Optimization
Show answer and example
Answer: No. Real rewards need correctness and optimization tests, independent sandbox calibration, and handling sparse advantages and stale batches. A reward formula is not execution or acceleration evidence.
Illustrative example: Synthetic speed scores test update plumbing without timing code.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why let only Off-Context GRPO behavior see privileged hints?
Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information
Show answer and example
Answer: Hints help collect successes while the target sees the original prompt as deployed. Letting target see the solution evidence would change the task.
Illustrative example: A sampler can see a draft; omit it when scoring the target.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why sample with hints but optimize and infer without them?
Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information
Show answer and example
Answer: Hints increase successful trajectories and shift behavior. Correct using unhinted target versus hinted behavior ratio. Dropping denominator or retaining hints at evaluation changes the claimed task.
Illustrative example: A training draft can help collect a solution; the student and test still see only the problem.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does Off-Context GRPO differ from SFT on hinted replies?
Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information
Show answer and example
Answer: It retains verifiable rewards and relative group signals and corrects hinted behavior versus unhinted target. Imitation doesn't automatically correct that shift. Both must isolate evaluation gold.
Illustrative example: A hint makes a path common without implying the unhinted model originally sampled it often.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
What contexts condition numerator and denominator in Off-Context GRPO?
Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information
Show answer and example
Answer: Numerator uses target probability on the original prompt; denominator uses actual behavior probability with privileged context. Same reply prefix, different contexts; don't add the hint to both.
Illustrative example: Strip hints only from target scoring, not from the sampled behavior's probability.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why freeze singular values but adjust singular frames in ISO?
ISO: An RLVR-Native Optimization Stack
Show answer and example
Answer: Fixed spectral strength and rotated input/output directions constrain adaptation without freezing the whole layer or keeping function unchanged.
Illustrative example: The same stretches in rotated bases can act on different features.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
How does fixed-spectrum ISO differ from ordinary low-rank updates?
ISO: An RLVR-Native Optimization Stack
Show answer and example
Answer: It adjusts input/output singular directions while retaining magnitudes, rather than adding arbitrary low-rank changes. This constrains reachable weights and structures expert merges. Small tests don't establish full model optimization.
Illustrative example: Equal singular values with rotated directions can implement different mappings.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Are ISO fixed spectrum and GCPO protected subspaces the same?
ISO: An RLVR-Native Optimization Stack
Show answer and example
Answer: No. ISO fixes values and changes frames; GCPO excludes updates into selected principal directions. They fix different quantities and allow different changes.
Illustrative example: Rotating directions at fixed scale isn't holding principal directions fixed and using their complement.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
How avoid training Σ in ISO?
ISO: An RLVR-Native Optimization Stack
Show answer and example
Answer: Keep Σ0 fixed and optimize only permitted input/output frames, checking structure. Free matrix updates change spectrum. Local candidate tests don't cover the full LLM stack.
Illustrative example: Unrestricted gradients on a reconstructed weight bypass fixed singular values.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why does a fixed ratio clip give uneven distance budgets in RIPO?
Show answer and example
Answer: Equal ratio changes have different geometric effects at different old probabilities. Fisher–Rao geometry adjusts bounds for more comparable local budgets.
Illustrative example: The same relative increase for rare and frequent actions need not move the distribution equally.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why isn't one PPO ratio interval the same policy distance at every old probability?
Show answer and example
Answer: Local geometry and KL cost depend on old probability. RIPO uses Fisher–Rao probability-dependent bounds, not different advantage labels. Actual update stability still needs verification.
Illustrative example: Doubling rare and common actions affects remaining mass differently.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How do adaptive RIPO bounds differ from raising global PPO ε?
Show answer and example
Answer: Global ε broadens every action. RIPO conditions bounds on old probability rather than uniformly becoming more aggressive, without guaranteeing universal stability.
Illustrative example: Exploration needs in low-probability regions need not relax frequent actions equally.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Should RIPO dynamic clipping read pre- or post-update probabilities?
Show answer and example
Answer: Set boundaries from frozen old-policy action probabilities while evaluating current ratios. Recomputing from current shifts the trust region during updates. Candidate tests aren't long-CoT training.
Illustrative example: Increasing an action's probability should not rewrite that round's original bound.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why convert some old-policy SIS tokens into usable current-policy samples?
Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment
Show answer and example
Answer: Asynchrony/reuse induce staleness and products of ratios have high variance. Rejection sampling reduces correction for accepted tokens; the host method handles rejected ones without dropping all data.
Illustrative example: Retain useful accepted positions within an old batch.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why does SIS set accepted ratios to one, and does this make the whole trajectory on-policy?
Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment
Show answer and example
Answer: Under ideal rejection sampling the accepted conditional action distribution is corrected at a fixed prefix, so no original ratio is needed there. Rejects follow host correction. Whole histories remain old, and top-K is approximate.
Illustrative example: Accepting one position never sets all trajectory ratios to one.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Is setting accepted token ratios to one the same as clipping to one?
Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment
Show answer and example
Answer: No. Acceptance is a distributional rejection-sampling event, not numerical capping. Exact envelopes and behavior top-K approximations have different guarantees.
Illustrative example: Replacing every large ratio with one gives no sampling guarantee.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Which policy supplies SIS top-K, and what happens to rejections?
Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment
Show answer and example
Answer: Approximate the envelope on behavior top-K. Set accepted ratios to one, keep GRPO/DAPO/GSPO correction for rejected tokens, rather than deleting or resetting them.
Illustrative example: One rejected token does not force dropping the full reply.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why focus STARE on surprising tokens rather than equally reinforce a trajectory?
Show answer and example
Answer: The same advantage can affect entropy differently by position. Select critical surprisal quantiles and adjust weights with target-entropy feedback, not terminal reward alone.
Illustrative example: An uncertain branch and punctuation can share reward without serving equal exploration roles.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why combine surprisal with target entropy rather than always reward surprising tokens?
Show answer and example
Answer: Surprising positions can influence entropy strongly; amplifying them always can overdisperse policy. Select batch quantiles, then adjust advantages based on entropy error. Surprise is not a correct-action label.
Illustrative example: A rare token can need opposite adjustments under an overconfident vs overrandom policy.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does STARE's entropy feedback differ from a fixed entropy bonus?
Show answer and example
Answer: Fixed bonuses always push one direction; feedback reacts to the current–target gap. Targets and measurement matter. Higher entropy is not inherently better and surprise doesn't imply correctness.
Illustrative example: Above-target entropy requires a different control direction from below-target entropy.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Why identify STARE positions by batch quantiles?
Show answer and example
Answer: Quantiles measure relative salience in the current batch, not an absolute threshold. Fix valid-token masks and probability source; exclude padding. Local checks cover weighting and feedback only.
Illustrative example: Artificial low padding probabilities can shift the critical quantile.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why represent one token and the remaining vocabulary as binary in KPop?
Show answer and example
Answer: A sampled ratio ignores absolute mass. Binary comparison includes both token and complement, using forward/reverse KL to gate train–serve mismatch.
Illustrative example: Equal multiplicative changes for rare and common tokens affect complement mass differently.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why compare both binary KLs rather than only bound a ratio?
Show answer and example
Answer: The same ratio means different movement at different token probabilities. The binary distribution captures token and complement, and KL asymmetry motivates both directions. Passing does not establish token correctness or system-wide lack of bias.
Illustrative example: The same relative shift for a rare and common word is not geometrically equivalent.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does bidirectional binary KL differ from a fixed ratio mask?
Show answer and example
Answer: KL depends on probability level, whereas a ratio mask uses relative multiples. Both directions must pass because KL is asymmetric. A mask doesn't validate synchronization of real asynchronous serving.
Illustrative example: Small KL one way can hide mismatch the other way.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Does KPop retain a token if either KL is small or only if both are?
Show answer and example
Answer: Both directions must be at or below threshold. Stabilize boundary probabilities and track serving vs training source. No local real train/inference engine or MoE cluster.
Illustrative example: Passing only train-to-serve KL still means mask out.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why compute GPRL group advantages per preference dimension?
Show answer and example
Answer: Dimensions have different units and variance. Normalize separately, then aggregate by task context. Normalization avoids accidental numerical dominance, not a claim of equal importance.
Illustrative example: A 100-point format scale would swamp binary correctness under raw summation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why normalize GPRL advantages by dimension before context aggregation?
Show answer and example
Answer: Raw sums let large scales or variance dominate and hide tradeoffs among helpfulness, format and brevity. Standardize, weight by context and monitor drift. Normalization isn't value prioritization; exact accuracy isn't a multi-dimensional evaluation.
Illustrative example: Separate a 100-point metric's units from how much users want it weighted.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does GPRL differ from summing rewards then applying GRPO?
Show answer and example
Answer: Summation mixes scales and conflicts before normalization. Separate standardization preserves each dimension's relative information before aggregation; the operations do not commute. Test context weights and drift separately.
Illustrative example: One high-variance dimension can dominate normalized totals while others barely matter.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Does local GPRL exact-answer regression refute the paper's open-ended results?
Show answer and example
Answer: No. Retain the negative accuracy result, but real human multi-dimensional preferences and open generation judging are absent. Track per-dimension mean, variance and effective weights.
Illustrative example: Brevity can improve while exact match worsens, so answer score alone isn't complete.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why use group InfoNCE for sequence preference in ConSPO?
Show answer and example
Answer: Multiple same-prompt sequences compete in one objective. Average logp reduces length scale effects and temperature controls separation.
Illustrative example: Boost a good solution relative to alternatives rather than score its likelihood alone.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why use length-normalized sequence scores for InfoNCE rather than cumulative token reward?
Show answer and example
Answer: Summed logp changes scale with length and can confuse verbosity with quality. Average per response then contrast within the same prompt. Temperature and normalization still affect gradients; relative quality is not absolute correctness.
Illustrative example: Two correct replies can differ in length; average before comparing so raw sum does not penalize the longer one.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does ConSPO group contrast differ from independent binary pairs?
Show answer and example
Answer: One denominator couples all candidates, unlike independent pair losses. Group size and negative composition change the objective.
Illustrative example: A new competitor changes the shared denominator, not just one pair.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Where do length normalization and temperature apply in ConSPO?
Show answer and example
Answer: Average valid-response logp, divide by temperature, then use stable group log-sum-exp. Exclude prompts/padding. Local kernel is not paper-scale training.
Illustrative example: Changing tokenwise temperature and denominator arbitrarily departs from sequence scoring.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why is Missing Old Logits more than a logging issue?
Show answer and example
Answer: Historical training-side probabilities separate train–inference mismatch from stale weights. Serving probabilities alone conflate both, making the correction ambiguous.
Illustrative example: Token drift can come from engine differences or several policy updates.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why can't the total ratio disentangle mismatch and staleness without old training logits?
Show answer and example
Answer: Current training vs old inference factors through old training probabilities. Missing that intermediate hides whether changes come from the engine or updates. Recover the anchor by snapshot or an old-logit model, not zero-fill.
Illustrative example: A changed score on an old sample cannot identify the original engine gap without its training snapshot.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Why do train–inference and staleness corrections need different denominators?
Show answer and example
Answer: One compares training and serving implementations of the same old policy; the other compares current and old training policies. Their product is the total ratio, not interchangeable terms.
Illustrative example: Changing inference engines at fixed weights can create mismatch without training staleness.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Can current logits simply replace missing old training logits?
Show answer and example
Answer: No. Snapshots, old-logit models, interrupted synchronization and EWMA are explicit alternatives with costs/assumptions. Local decomposition and state tests are not cluster deployment.
Illustrative example: Rescoring old samples with updated weights rewrites the baseline rather than restoring history.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why examine response-length trajectories in LUSPO?
Show answer and example
Answer: RLVR objectives implicitly weight length differently, potentially causing collapse or drift. Normalize sequence logp as prescribed to separate reward learning from length scale.
Illustrative example: Stable accuracy with shrinking replies can still lose long-reasoning coverage.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why can sequence-level LUSPO updates still need length-bias checks?
Show answer and example
Answer: Aggregation does not remove length coupling in cumulative logp. LUSPO reorganizes scores to avoid systematic length pressure. Short-task normalization checks do not validate long-CoT dynamics.
Illustrative example: Two replies with similar per-token changes can have different cumulative ratios solely because of length.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does the LUSPO sequence ratio differ from tokenwise clipping?
Show answer and example
Answer: Exponentiate length-normalized sequence log-ratio and share the signal over the response. Tokenwise clips act locally, with different dynamics. Sequence aggregation alone doesn't guarantee long-length stability.
Illustrative example: An increased token ratio and a decreased one can cancel in a sequence but clip differently.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Why don't local short arithmetic tasks establish long-CoT length stability?
Show answer and example
Answer: Generation, sequence ratios and normalization run, but short replies do not cover long reasoning dynamics. Use effective response length without padding/prompt.
Illustrative example: A dozen-word answer doesn't test drift across thousands of tokens.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Does calibration-aware RL score the entire explanation through its decision token?
Show answer and example
Answer: No. It studies final-decision probability and calibration, not factual fidelity of all reasoning. Correct choices and credible probabilities require separate measurement.
Illustrative example: High probability for A doesn't make every preceding explanation step correct.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why constrain decision probability if accuracy improves while ECE worsens?
Show answer and example
Answer: Correctness only needs the top class right; calibration requires confidence to match actual hit frequency. RLVR can make correct choices overconfident. Measure separately, choose T on validation and reserve test. Post-hoc calibration is not full RL reproduction.
Illustrative example: Two models with identical labels can assign near-certainty or moderate confidence and differ in probability quality.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does local temperature calibration differ from full RLVR training?
Show answer and example
Answer: Temperature rescales existing logits; full training changes the decision model. Shared calibration goals do not make postprocessing a full reproduction.
Illustrative example: A frozen model with tuned T uses no resampled trajectories or policy gradient.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
May calibration T be selected on test NLL?
Show answer and example
Answer: No. Fit validation NLL, freeze, then report test accuracy and calibration. Choosing on test biases final evaluation. Local coverage is calibration and split isolation.
Illustrative example: Trying all temperatures on test and reporting the best overstates generalization.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why consume each Online IcePop rollout batch once?
Show answer and example
Answer: One update avoids extra staleness from reuse, making initial policy ratio one. Train/inference numerical correction remains.
Illustrative example: Discard the fresh batch after its update instead of optimizing against it again.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why retain engine correction when Online IcePop updates only once?
Show answer and example
Answer: Fresh one-use rollouts eliminate reuse staleness, so policy ratio at sampling is one and PPO clipping unnecessary. Equal weights can still produce different probabilities in training vs inference; IcePop keeps its two-sided mask and in-range ratio.
Illustrative example: A fresh answer can come from a lower-precision engine; freshness is not engine equality.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Which correction does Online IcePop remove?
Show answer and example
Answer: Remove reuse-induced stale-policy ratio and PPO clip, not the two-sided engine mask or original in-range ratio. Repeated updates invalidate the online simplification.
Illustrative example: Scheduling must enforce one use; deleting a line doesn't make training online.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
How verify Online IcePop never silently reuses a batch?
Show answer and example
Answer: Track unique batch IDs and consumption; reject a second update. A local candidate group is only a scheduling proxy, not distributed rollout infrastructure.
Illustrative example: Retried delivery of the same batch is not fresh data.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why model both train/inference mismatch and staleness in miniRL?
Show answer and example
Answer: Equal weights may score differently across engines; old/new weights drift through learning. Separate them for appropriate on/off-policy correction.
Illustrative example: A kernel change and a policy update are distinct causes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why split engine mismatch and policy staleness, and where does routing replay help?
Show answer and example
Answer: Numerical engine differences and weight drift are distinct. Correct on-policy probabilities, combine off-policy updates with clipping and replay rollout MoE routing to reduce path mismatch. Both probabilities and paths need testing; local kernels aren't a cluster.
Illustrative example: One token may route to different experts even with synchronized weights, which a stale-policy ratio does not fix.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
What do MoE Routing Replay and ratio clipping address?
Show answer and example
Answer: Replay restores expert execution paths; clipping limits update magnitude from probabilities. Clipping cannot reconstruct old routes.
Illustrative example: Saving which expert executed is different from lowering a large ratio.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Does the local miniRL kernel prove distributed MoE stability?
Show answer and example
Answer: No. It checks correction, clipping and routing state semantics. Real replay needs recorded token routes and actual aligned execution, not a label.
Illustrative example: A replay flag doesn't prove forward passes used the recorded experts.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why does IcePop mask very small train-to-serve ratios?
Show answer and example
Answer: Upper clipping only controls huge weights; tiny ratios may also signal mismatch. A two-sided interval keeps trustworthy ratios without snapping others to boundaries.
Illustrative example: A training probability far below serving can be unreliable even though its weight is small.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
How is zeroing out-of-range IcePop updates different from clipping to a boundary?
Show answer and example
Answer: Clipping preserves boundary-weighted contribution; masking drops the current token gradient. IcePop uses two fixed bounds and original in-range correction. It filters mismatch, not text quality.
Illustrative example: Tiny ratio is unaffected by upper clipping, but two-sided masking excludes it instead of raising it to the lower bound.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How do IcePop masking and TIS clipping treat out-of-range samples?
Show answer and example
Answer: IcePop zeroes their gradient; TIS caps high weights but keeps them contributing. IcePop uses original in-range ratios, not ones.
Illustrative example: An oversized ratio can retain capped contribution in TIS but none in IcePop.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Must IcePop distinguish engine ratio from policy ratio?
Show answer and example
Answer: Yes. Engine ratio measures implementation mismatch; policy ratio measures current vs sampling policy. Mask the specified engine ratio only. No local real MoE/asynchronous system.
Illustrative example: Using model-learning drift as the engine mask could drop normal probability changes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Which PPO gradients does GPPO recover?
Show answer and example
Answer: Ordinary clipping can stop gradients for positive-advantage high ratios and negative-advantage low ratios. GPPO restores backward signal while keeping the forward value.
Illustrative example: A bad action already made rarer may still need an avoidance gradient.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why can GPPO have PPO-like forward values and nonzero clipped-region gradients?
Show answer and example
Answer: Stop-gradient constructions decouple clipping values and derivatives. Boundary weights restore learning in flattened quadrants for positive exploration and negative examples. Inspect gradients, not just loss.
Illustrative example: Two operators can print one clipped score but only one has a boundary-weighted derivative.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Why can identical forward losses produce different updates?
Show answer and example
Answer: Stop-gradient can change derivatives without changing values. GPPO separates forward and backward, so scalar logs cannot verify it.
Illustrative example: Two functions with equal output can have different derivatives through detach.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Is equality of clipped GPPO/PPO loss values enough to test implementation?
Show answer and example
Answer: No. Verify gradients in out-of-bound advantage quadrants against PPO's zero gradients, and explicit detach boundaries. No paper-scale long-CoT training.
Illustrative example: Positive advantage and a ratio above the upper bound directly test the recovered path.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why use TIS for equal-weight training and inference engines?
Show answer and example
Answer: Precision and kernels can change probabilities, so rollout and training rescoring distributions differ. The ratio corrects implementation mismatch, not task rewards.
Illustrative example: Two compute paths using identical weights can assign slightly different token likelihoods.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why upper-clip engine ratios but keep small weights in TIS?
Show answer and example
Answer: Correction targets the training distribution. Huge weights can dominate variance; cap them. Small ratios indicate smaller target contributions, not automatic invalidity. This is not PPO policy ratio or two-sided masking.
Illustrative example: A token common in rollout but rare in training should contribute less without necessarily being removed.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Why does TIS upper-clip instead of two-sided dropping like IcePop?
Show answer and example
Answer: Capping large weights controls heavy tails while retaining small contributions. Two-sided masking drops severe mismatch in either direction. Zeroing small TIS weights changes the method.
Illustrative example: A tiny TIS ratio still contributes a little.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Which probability is the TIS denominator?
Show answer and example
Answer: Use the actual rollout engine probability below corresponding training probability; a separate policy ratio may follow. Local deterministic perturbations aren't vLLM/FSDP validation.
Illustrative example: Reversing the ratio can upweight an example that should be downweighted.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why use one importance ratio for a full response in GSPO?
Show answer and example
Answer: Terminal reward is sequence-level, and token outliers can cause fragmented clipping. Average token log ratios to form one response weight and avoid local outlier dominance.
Illustrative example: One rare token's large shift need not determine distinct clips throughout a long reply.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why average token log ratios before exponentiation and clipping?
Show answer and example
Answer: One outcome belongs to one answer. One geometric weight gives all tokens the same relative advantage and clipping decision despite local noise. It changes credit aggregation and doesn't eliminate all length/MoE mismatches.
Illustrative example: A correct answer with one anomalous ratio gets different credit under token clipping and shared weighting.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Is GSPO's geometric-mean ratio the actual sequence probability ratio?
Show answer and example
Answer: No. GSPO exponentiates the mean log ratio; the exact sequence ratio exponentiates the sum, multiplying token ratios. Averaging changes length scaling and the objective.
Illustrative example: Repeated small shifts accumulate in a product but the geometric mean measures typical shift.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Should GSPO clip per token or per response?
Show answer and example
Answer: Clip one ratio per response, shared across tokens. Mask padding from length. Candidate experiments do not reproduce real token rollouts, MoE routes or clusters.
Illustrative example: Clipping each token and averaging usually differs from averaging then clipping.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why predict both an answer and confidence in RLCR?
Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty
Show answer and example
Answer: Correctness-only reward does not incentivize honest probabilities. A proper score evaluates probability and outcome, rewarding truthful probabilities in expectation under its assumptions.
Illustrative example: A model that often errs should report moderate confidence instead of certainty.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why need proper scoring in addition to correctness rewards?
Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty
Show answer and example
Answer: Binary reward cannot distinguish justified caution from false certainty. Proper scoring couples probabilities with outcomes to incentivize honesty. Report accuracy and calibration separately; local Brier gradients aren't full RL.
Illustrative example: A quiz bot claiming certainty on unfamiliar items can be penalized for that confidence even when ordinary rewards ignore it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does RLCR differ from displaying confidence after training?
Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty
Show answer and example
Answer: A displayed field needn't affect optimization; RLCR incorporates a scoring rule. Local Brier/hybrid objectives are mechanism evidence, not full rollouts.
Illustrative example: A 90% badge doesn't mean bad calibration was penalized in training.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Which classes receive local RLCR Brier gradients?
Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty
Show answer and example
Answer: Compute squared differences between the complete softmax distribution and one-hot target and backpropagate over all candidates, not just the displayed choice. Hybrid includes CE; no paper-checkpoint RL.
Illustrative example: Unselected classes consume probability mass and contribute to scoring.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why clip importance weights in CISPO?
Show answer and example
Answer: Bound magnitude while preserving advantage direction and learning, unlike PPO min-surrogate flat regions. Separate frozen weights from differentiable logp.
Illustrative example: A capped positive weight can still reinforce through logp.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why does CISPO retain gradients for some tokens PPO would clip flat?
Show answer and example
Answer: Clipped weights scale log-probability gradients while retaining advantage direction. PPO's minimum is flat in certain quadrants. Test gradients, not only values or clipped advantages.
Illustrative example: A rare negative-advantage token can still receive bounded suppression even if PPO is flat.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Why aren't CISPO and PPO the same because both clip?
Show answer and example
Answer: PPO takes the minimum of clipped and unclipped surrogates. CISPO multiplies logp and advantage by clipped weight. Backward and out-of-range behavior differ despite matching bounds.
Illustrative example: One boundary can stop gradients; the other preserves updates.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
What if CISPO differentiates directly through the clipped ratio?
Show answer and example
Answer: Extra weight derivatives change the policy-gradient objective. Keep rollout, weight construction and logp path separate. Local checks do not reproduce MiniMax training.
Illustrative example: Current action logp should supply gradient without duplicating it through the weight branch.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Where does TTRL get rewards on unlabeled test tasks?
Show answer and example
Answer: Sample several answers, use majority consensus as a pseudo-label, reward agreement and update online. No gold, but consensus can be wrong; evaluate separately.
Illustrative example: Repeatedly guessing the same wrong number creates agreement, not truth.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why label TTRL as adaptation rather than fixed-model evaluation?
Show answer and example
Answer: Test inputs trigger sampling and immediate parameter updates, so the evaluated model changes. Report sampling/adaptation budgets and keep gold in independent scoring. Pseudo-label consensus can reinforce errors; unlabeled isn't no learning.
Illustrative example: Sampling and updating before answering differs from one call to the original model even without gold.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does TTRL differ from majority-vote inference?
Show answer and example
Answer: Voting just aggregates outputs; TTRL updates parameters using pseudo-rewards and affects later behavior. Account for training compute and persistent state.
Illustrative example: After the same samples, voting returns; TTRL additionally optimizes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
May TTRL answer normalization see test gold?
Show answer and example
Answer: No. Equivalence and votes must use permitted output rules; gold is scored afterward. Local loops aren't paper-scale real rollouts.
Illustrative example: Do not filter by the correct number before claiming an autonomous vote.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why combine offline demonstrations and online exploration in LUFFY?
Show answer and example
Answer: Demonstrations reveal paths students rarely sample; online rollouts preserve current exploration. Handle shared support and source probabilities rather than pure imitation or narrow exploration.
Illustrative example: Experts show a proof while the student still generates attempts and receives feedback.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why track behavior μ in mixed demonstration/rollout batches?
Show answer and example
Answer: Shared batch doesn't make sources on-policy. Importance ratios and regularization address source shift while keeping exploration. Students must still reach uncovered states; do not leak demos into evaluated inputs.
Illustrative example: Record expert success separately from failed student rollouts, not as a student sample.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does LUFFY differ from inserting demonstrations into an ordinary GRPO group?
Show answer and example
Answer: External demonstrations have another sampling source and need correction/regularization. Treating them as student samples gives wrong denominator. Source-aware updates are more than adding good text.
Illustrative example: An inserted expert solution is not a lucky current-policy sample.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
What roles do μ and ref have in LUFFY?
Show answer and example
Answer: μ describes mixture sampling; ref anchors KL. Track online vs demonstration provenance. Local candidates aren't paper text/distributed training.
Illustrative example: Expert data and initial-student reference need not have the same probabilities.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why emphasize a critic for long reasoning in VAPO?
Show answer and example
Answer: Sparse terminal rewards leave prefixes without direct feedback; a critic supplies a baseline. Bias can grow with length, motivating pretrained value and length adaptation.
Illustrative example: Estimate later returns before an intermediate reasoning step gets terminal reward.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why repair critic quality and adapt GAE to length instead of dropping the critic?
Show answer and example
Answer: Sparse rewards and varying horizons amplify value error. VAPO pretrains the critic and adapts advantage estimation and actor to length. It doesn't remove value learning, and local proxy lengths aren't long-CoT evidence.
Illustrative example: Short and long solutions receive reward at different delays, so one estimator can have different bias/variance across them.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
What does VAPO trade for critic-free GRPO?
Show answer and example
Answer: GRPO uses same-prompt relative rewards without a critic. VAPO trains a critic for prefix values. Their costs and error sources differ; fewer models isn't higher capability.
Illustrative example: An all-wrong group has no relative signal while a critic can still estimate, possibly incorrectly.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Why isn't the local VAPO critic a reproduced pretrained value model?
Show answer and example
Answer: Candidate actions and proxy lengths simulate value–length coupling, not real long prefixes. Retain response-dependent GAE λ rather than claiming a fixed λ is equivalent.
Illustrative example: A table of candidate values isn't a neural critic reading thousands of reasoning tokens.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why audit both response-length averaging and group standard deviation in Dr. GRPO?
Show answer and example
Answer: They change weights by length and problem difficulty, not just numerical stability. Remove both while retaining group mean centering so reward differences preserve their scale.
Illustrative example: Equal reward gaps get different magnitudes after normalization when group variances differ.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why remove both group SD and within-response averaging?
Show answer and example
Answer: SD reweights difficulty groups by their local variation; token averaging changes long-versus-short contribution. Keep centered rewards and fixed aggregation, not remove all normalization or promise universal gains.
Illustrative example: A tiny group SD can amplify a comparable reward gap; response length adds another weight.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Does Dr. GRPO still center group rewards like GRPO?
Show answer and example
Answer: Yes. Subtract same-prompt mean, but omit the specified length and SD normalization. Dropping SD does not remove the baseline, and adding a global SD would change the target.
Illustrative example: Judge each reply relative to group mean without dividing by group variation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Can proxy lengths establish removal of long-CoT length bias?
Show answer and example
Answer: No. A candidate policy has no generated length distribution. Proxy lengths expose normalization differences, not real token losses or long responses.
Illustrative example: Changing a simulated length doesn't produce early stopping behavior.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why dynamically skip all-correct and all-wrong DAPO groups?
Show answer and example
Answer: Binary rewards without variance give no relative advantage and waste batch slots. Sample groups with distinctions; this doesn't solve exploration on every difficult problem.
Illustrative example: Four failures give no ordering, so resample rather than repeatedly train on zeros.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why drop all-correct groups in Dynamic Sampling?
Show answer and example
Answer: Centering equal all-correct/all-wrong rewards yields no relative direction, not worthless replies. Resample differentiable groups; Clip-Higher, token aggregation and length shaping handle other effects. Never select questions on test.
Illustrative example: A mixed-success group provides group-relative learning absent from an equal-reward group.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Which side does DAPO Clip-Higher relax?
Show answer and example
Answer: Raise the upper bound to retain room for beneficial probability increases; lower bound remains separate. Dynamic sampling, token loss and overlength shaping are also defining pieces.
Illustrative example: Easier increase of a useful action needn't loosen downward changes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
How does DAPO token-level normalization differ from per-response averages?
Show answer and example
Answer: Sum over valid response tokens, then normalize together. Per-response means weight lengths differently. Also preserve dynamic filtering and overlength penalty. Proxy lengths don't reproduce real masks or 32B training.
Illustrative example: Averaging a short and long reply separately is not averaging every token together.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why avoid each prompt's SD in REINFORCE++?
Show answer and example
Answer: Few samples make local variance noisy and can overamplify some difficulty groups. Keep group centering and use a global scale without a critic.
Illustrative example: An accidentally tiny variance should not give one prompt excessive weight.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why center locally but scale globally?
Show answer and example
Answer: Group mean removes common difficulty; local SD may amplify incidental low variance. Cross-batch scale makes units consistent without critic. Window and distribution matter; small EMA tests don't prove stability.
Illustrative example: Two similarly separated groups shouldn't receive very different weights just because one has tiny variance.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Why isn't global EMA normalization a reordering of GRPO normalization?
Show answer and example
Answer: Statistics use different domains. Group means compare same-prompt samples; global second moments share scale. This changes cross-problem weighting.
Illustrative example: Groups with the same gap but different variances differ under local division, not a shared scale.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Can REINFORCE++ reestimate EMA from evaluation gold?
Show answer and example
Answer: No. Maintain training state in batch order; test answers cannot tune scale. Small-candidate EMA validates semantics, not the large-batch distribution.
Illustrative example: Scanning all test rewards to set the denominator leaks evaluation information.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why does RLOO treat a full response as one action?
Show answer and example
Answer: Terminal RLHF rewards suit response-level REINFORCE and avoid prefix values. Other same-prompt samples form a variance-reducing baseline; multiple samples are still needed.
Illustrative example: A summary's terminal score weights its complete logp relative to peers.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why exclude the current response from RLOO's same-prompt baseline?
Show answer and example
Answer: Peers provide an independent reference, not the current action's random reward. Treating a response as one action avoids a value network but still requires enough samples, consistent rewards and any specified reference/evaluator.
Illustrative example: For reply one of four, average the other three so the baseline doesn't move with its own reward.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How does leave-one-out differ from a mean including the current sample?
Show answer and example
Answer: LOO excludes the sample's own reward from its baseline. Including it changes dependency and scaling. Both are critic-free, but group size matters.
Illustrative example: With two replies, the first's LOO baseline is the second, not both averaged.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Can RLOO use the same formula at K=1?
Show answer and example
Answer: No. K−1 is zero and there are no peers. Require sufficient group size and KL-shaped returns as appropriate. Local K=4 candidates aren't 7B generation and real RM.
Illustrative example: Zero-filling an empty baseline changes the estimator.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why can GRPO omit a policy-sized value model?
Show answer and example
Answer: Same-prompt group mean and SD estimate relative advantages instead of a learned prefix value. Save critic resources but require repeated rollouts and differing feedback.
Illustrative example: Better attempts receive positive advantage relative to peers.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
How does GRPO obtain advantages without critic, and what are old and reference for?
Show answer and example
Answer: Center and scale group rewards for relative signal. Old gives sampling probabilities for ratio/clipping; reference anchors KL. Equal rewards provide no contrast. Candidate updates don't reproduce full math training.
Illustrative example: A reference change alters regularization, not the historical rollout source.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Why are GRPO old and reference policies different roles?
Show answer and example
Answer: Old supplies rollout and proximal ratios, reference anchors long-term KL. Old can refresh each batch independently of the reference; mixing couples update limits and regularization.
Illustrative example: Fresh sampling updates old policy, not necessarily initial reference.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
What should GRPO do with equal group rewards?
Show answer and example
Answer: Centered advantages are zero and SD needs protection; never invent differences using evaluator gold. Local six-action candidates aren't DeepSeekMath generation.
Illustrative example: Don't rank four incorrect attempts by gold proximity unless the reward protocol actually says so.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
Why use same-prompt greedy reward as ReMax baseline?
Show answer and example
Answer: It gives a prompt-specific reference without training value. Reinforce samples above greedy and suppress below it; greedy need not be correct.
Illustrative example: A sampled explanation better than the usual first choice gives positive credit.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Must the ReMax greedy baseline be correct?
Show answer and example
Answer: No. It is a variance-reducing comparator for the same prompt. Relative returns can be positive or negative independent of absolute correctness; baseline isn't gold. Both paths still cost generation and scoring.
Illustrative example: A wrong greedy and correct sample yield positive signal; two wrong replies do not make greedy the standard answer.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
How do ReMax and RLOO baseline sources differ?
Show answer and example
Answer: ReMax additionally executes greedy decoding. RLOO uses other randomized responses. Both save the critic but spend generation differently and have different baseline errors.
Illustrative example: Compare one greedy rollout's cost to multiple peer samples.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Should ReMax differentiate through greedy baseline reward?
Show answer and example
Answer: No. Treat it as fixed return, not a gradient through discrete decoding or scoring. Use the same prompt/protocol for both. Local greedy candidates do not establish memory or latency.
Illustrative example: Do not invent an argmax reward gradient while updating the sampled policy.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Basics
What supervision enters each InstructGPT stage?
Show answer and example
Answer: Human demonstrations train SFT; human rankings train a reward model; PPO optimizes reward with reference KL. Models and data differ, so a PPO loss alone is not the pipeline.
Illustrative example: Having demonstrations without rankings doesn't mean a reward model was trained.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Core mechanism
Why distinguish reward model, critic, old policy and reference?
Show answer and example
Answer: Reward model scores preferences, critic predicts returns for advantages, old policy records sampling, and reference anchors SFT drift. Shared initialization doesn't merge roles. Candidate updates aren't human ranking, reward-model training or large-scale RLHF.
Illustrative example: A reward model rates a reply, a critic forecasts its return, and reference measures deviation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Method comparison
Why distinguish PPO clipping from reference KL?
Show answer and example
Answer: Clipping limits a proximal step relative to old policy; KL anchors drift relative to a reference over longer horizons. They aren't substitutes and don't prevent reward exploitation.
Illustrative example: Repeated small steps can still travel far from the initial policy.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Policy optimization: which policy samples and which policy learns?
Implementation details
Does local PPO-RLHF include a learned reward model?
Show answer and example
Answer: No. Deterministic outcome/process axes reward a linear candidate policy. Actor, old policy, critic, clip and KL structure do not imply GPT-3 training or human data.
Illustrative example: A numerical critic can verify baseline updates without reproducing RLHF.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Topic guide and reading order
Start with this question¶
Which quantities must remain distinct across online sampling, old-policy probabilities, and current-policy updates?
Takeaway¶
RL post-training collects trajectories and rewards, then computes advantages and an update objective. GRPO uses relative rewards for a group of answers to one prompt instead of a separate critic. DAPO changes clipping, uninformative groups, and length handling. Asynchrony, off-policy replay, and inference/training engine differences change the sampling distribution. Old, current, and reference policies play different roles; their probabilities are not interchangeable.
Example¶
For rewards 0, 0, 1, 1 on four answers, within-group advantages give a direction. If all four fail and get the same reward, group centering cannot distinguish them. Replaying cached trajectories does not freeze current probabilities, so do not blindly reuse old advantages and density ratios. Check whether importance weights are required and which sampler they refer to.
Common pitfall¶
Stability and capability require separate evidence. Finite gradients, clipping tests, and ratio checks do not prove better reasoning. Low offline loss cannot replace actual rollout comparisons. Full checkpoint and long-trajectory runs retain the cost and GPU gates; a runnable formula is not finished training.
Reading order¶
Start with GRPO sampling, advantages, ratios, and reference KL. Then read DAPO for zero-variance groups and training stability, and TIS for probability conventions across the sampling and training engines.
Check your understanding¶
If every answer in a group has the same reward, can a larger batch or repeated calculation of the same advantages create a new direction?
Show suggested answer
Not from the same centered rewards. Check whether reward is too coarse, tasks are too easy or hard, or resampling or another legitimate signal would help. Do not read test answers to manufacture nonzero advantages. Recheck the objective and budget after any change.