Preference and alignment: identify the feedback first¶
简体中文 · English
LLM post-training / 100 questions · 25 papers
Try answering each question before opening the explanation. The bank grows as papers and questions are added.
Read this topic's comparisons, analysis, and cross-paper Q&A
100 questions · 25 papers
LLM post-training / Preference and alignment: identify the feedback first
Basics
Why distinguish two preference-flip rates rather than estimate one overall error rate?
CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF
Show answer and example
Answer: True positives flipped negative and true negatives flipped positive can have different causes. Estimating each conditional rate corrects observed-label risks separately; one error rate hides asymmetry.
Illustrative example: A length-biased annotator may reject correct short answers at a different rate than they accept incorrect long answers.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why match semantics as well as recovered preference, and may negative risk be clipped?
CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF
Show answer and example
Answer: Matching preference proportions alone can align semantically unrelated examples. Joint optimal transport uses representation distances and recovered preferences to estimate both noise rates. Fixed corrected loss can be negative; clipping changes its expectation, and inaccurate rates also break unbiasedness.
Illustrative example: Two domains can have half positive labels while one flips them often; use semantic representations to expose the difference.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
How does freezing the CleanRM noise estimator differ from jointly updating it with the reward model?
CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF
Show answer and example
Answer: A frozen estimator defines a fixed corrected risk. Joint updates move its coefficients and can lower loss by changing the explanation of noise. This is a different objective without the same fixed-rate risk interpretation.
Illustrative example: Estimate label flips before training; don't reinterpret temporarily misclassified samples as noisier.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
Should CleanRM clamp recovered labels or corrected losses to their conventional ranges?
CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF
Show answer and example
Answer: No. Recovered preferences can fall outside [0,1] and corrected risks are signed. Clamping changes the debiasing objective. Require rates summing to less than one and freeze the estimator during RM updates.
Illustrative example: A valid negative corrected loss should not be reset to zero for prettier curves.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Does GAW-PO label complete replies or each token as preferred/rejected?
GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
Show answer and example
Answer: Labels compare two replies to one prompt; there is no per-token annotation. Rejected replies can contain good steps or shared wording. Gradient alignment supplies credit at a finer granularity, not additional human token truth.
Illustrative example: If both answers start with the correct equation and differ only at the end, the rejection does not label the shared equation wrong.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why penalize rejected-response tokens less if they support the preferred direction?
GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
Show answer and example
Answer: A whole response's rejection does not condemn all tokens. Smaller negative weights for aligned gradients avoid suppressing shared useful steps. This changes credit, not factual verification.
Illustrative example: If both openings are right and only the rejected conclusion is wrong, an aligned opening needn't receive the same penalty.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
How does GAW-PO gradient alignment differ from weighting by token likelihood?
GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
Show answer and example
Answer: Likelihood measures how often the current model produces a token. Alignment measures whether penalizing it conflicts with preferred updates. Common tokens can be irrelevant and rare tokens helpful; this is a mechanism comparison, not an empirical ranking.
Illustrative example: Do not punish a rare term merely for low probability if it supports a correct explanation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
How does GAW-PO's positive-part cosine handle aligned, orthogonal and opposite gradients?
GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
Show answer and example
Answer: Weight is one minus positive cosine. Perfect alignment approaches zero; orthogonal/opposite directions keep weight one; partial positive alignment reduces penalty. Actual gradients are required. Local evidence is only the weight kernel.
Illustrative example: A negative cosine is first set to zero, so the weight does not exceed one.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Does COMPO's black-box optimization change prompts instead of weights?
A Zeroth-Order Paradigm for LLM Preference Alignment
Show answer and example
Answer: No. It updates model parameters without backpropagating through the preference objective. It needs parameter perturbation and an objective comparator. Prompt-only remote access does not satisfy that requirement.
Illustrative example: Load and compare two nearby checkpoints; editing chat prompts is another search.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
How does COMPO infer a direction from paired perturbations, and why threshold it?
A Zeroth-Order Paradigm for LLM Preference Alignment
Show answer and example
Answer: Which of positive and negative perturbations performs better on the same objective gives a directional sign. One-bit feedback is noisy, so coordinate thresholds suppress unreliable components. Objective sampling and evaluation still cost resources.
Illustrative example: Compare small forward/backward perturbations under fixed preferences; leave unreliable coordinates unchanged.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
What information do COMPO's zeroth-order comparisons and first-order backpropagation need?
A Zeroth-Order Paradigm for LLM Preference Alignment
Show answer and example
Answer: First order needs a differentiable objective and backward access; zeroth order ranks perturbed evaluations. Repeated comparisons still cost compute and noise affects direction; no backward pass does not imply free or more accurate.
Illustrative example: A preference judge can rank two variants without returning gradients for any layer.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
Why threshold each coordinate after a COMPO one-bit update?
A Zeroth-Order Paradigm for LLM Preference Alignment
Show answer and example
Answer: One comparison gives coarse information and weak coordinates may reflect noise. Thresholding suppresses them but can remove small genuine signals. Local tests cover comparison oracle and threshold kernel, not LLM alignment.
Illustrative example: A coordinate consistently near zero may be suppressed, not automatically deemed useless.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Does Style-Debiased DPO update knowledge or enforce one writing style?
Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Data
Show answer and example
Answer: Synthetic preferences should update knowledge; stylistic differences are confounders. Concision or ornament is not factual truth. That distinction motivates factual checks and group balancing rather than stylistic personalization.
Illustrative example: The same knowledge may be expressed in a sentence or a paragraph; both should stay correct.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现对应. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why reverse a preference if the rejected answer is factual?
Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Data
Show answer and example
Answer: Synthetic preferences can mislabel stylistic differences as knowledge errors, causing ordinary DPO to suppress correct expressions. Identify those replies and reverse their preferences with group-ratio weights to cancel style gradients. Reliable factual assessment and grouping are prerequisites.
Illustrative example: If both replies are factual but one is more courteous, directly following the label may teach courtesy rather than knowledge.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现对应. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
How does SD-DPO reversal differ from dropping stylistic pairs?
Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Data
Show answer and example
Answer: Dropping removes their contribution. Reversing with group-proportion weighting seeks to cancel an existing style direction. Factual correctness must be assessed first, or reversal could promote wrong answers.
Illustrative example: Check both formal and colloquial variants for correctness before balancing their style signals.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现对应. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
Do local SD-DPO factual scores establish that a real judge recognizes truth?
Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Data
Show answer and example
Answer: No. Fixed inputs supply the scores. The code tests reversal and group-balanced loss under given labels, not automated factual judging or knowledge gains.
Illustrative example: If the test declares an answer factual, correct reversal validates the rule, not identification of the fact.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现对应. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Is Normalized DPO's margin a human score gap or a model-relative preference?
Disentangling Optimization Scale from Preference Scale in DPO
Show answer and example
Answer: It is the model-side value in the loss. Data identify the winner; policy and reference probabilities determine the margin and beta sets its scale in the preference curve. Normalization does not rewrite labels.
Illustrative example: A human says A beats B without saying by three points; current probabilities still give a margin.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why can dividing by beta change gradients without moving Normalized DPO's optimum?
Disentangling Optimization Scale from Preference Scale in DPO
Show answer and example
Answer: At fixed positive beta, subtracting a constant and multiplying by a positive scalar preserve optima. Ordinary DPO couples preference-curve scale with update size; dividing centered softplus by beta removes an extra beta in the margin gradient.
Illustrative example: Reducing beta can weaken ordinary updates; normalization separates curve scale from step-size effects.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
Is Normalized DPO equivalent to changing the learning rate?
Disentangling Optimization Scale from Preference Scale in DPO
Show answer and example
Answer: Centering and dividing by beta rescales loss gradients. Learning rate scales the optimizer's update. They can be related under simple conditions, but momentum/adaptive statistics can change the trajectory.
Illustrative example: A second-moment optimizer need not maintain identical state under external step-size and loss scaling.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
How can normalized DPO avoid zero-over-zero for tiny beta?
Disentangling Optimization Scale from Preference Scale in DPO
Show answer and example
Answer: Compute centered softplus minus log 2 stably or use its small-beta limit. The gradient is negative sigmoid and does not vanish from an extra beta factor. Test the limit rather than substituting zero in the quotient. Candidate checks are not full training.
Illustrative example: Confirm finite limiting loss and gradient instead of catching exceptions and returning zero.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Does GRIP produce one merged model or run thinking and instruct models on every request?
GRIP: Granular Reward-Guided Parameter Interpolation for Efficient Reasoning
Show answer and example
Answer: It interpolates corresponding aligned parameters into one model and runs that model. It does not select experts online or generate two answers and vote. Arbitrary architectures cannot be combined without matching weights.
Illustrative example: Combine same-shape layer weights before one forward pass, not two responses followed by length selection.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why learn fine-grained interpolation instead of one ratio for thinking and instruct weights?
GRIP: Granular Reward-Guided Parameter Interpolation for Efficient Reasoning
Show answer and example
Answer: Layers and parameters serve different capabilities, so a uniform mix can weaken reasoning and brevity. Reward-guided coefficients vary across weights. The local probability interpolation is a proxy, not the paper's weight-combination validation.
Illustrative example: Preserve computation habits while reducing redundant explanation by using different coefficients across parameter regions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
How does GRIP interpolation differ from lowering inference temperature?
GRIP: Granular Reward-Guided Parameter Interpolation for Efficient Reasoning
Show answer and example
Answer: Weight interpolation changes distributions across contexts. Temperature rescales existing logits for one forward pass. Both can change length without representing the same mechanism or combining learned capabilities.
Illustrative example: Merging can change the favored reasoning step; lower temperature generally emphasizes already higher logits.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
What shape and evaluation constraints apply to GRIP interpolation?
GRIP: Granular Reward-Guided Parameter Interpolation for Efficient Reasoning
Show answer and example
Answer: Weight tensors and coefficients must align at the stated granularity; broadcasting must match layer or parameter semantics. Rewards must follow the allowed data protocol. A deterministic candidate diagnostic does not prove paper-checkpoint reasoning or length gains.
Illustrative example: Do not broadcast layer coefficients as vocabulary-column coefficients merely because multiplication works.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
What do PTO's dialogue policy, virtual user and oracle do, and which is updated?
Preference Tree Optimization: Enhancing Goal-Oriented Dialogue with Look-Ahead Simulations
Show answer and example
Answer: The policy proposes replies, the virtual user continues possible interactions, and the oracle evaluates expanded branches. Their comparisons label preferences for the current state and update only the dialogue policy. Simulated users and evaluators provide training signals, not real user behavior or joint updates.
Illustrative example: A consultation agent asks two possible questions, the simulator answers each, and the oracle compares progress. Train the agent's current reply selection.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why simulate later dialogue before labeling a current PTO response?
Preference Tree Optimization: Enhancing Goal-Oriented Dialogue with Look-Ahead Simulations
Show answer and example
Answer: A pleasing immediate reply may harm long-term goals. Compare rolled-out user interactions and turn branch-level differences into current-response DPO preferences. Depth changes cost and horizon; simulator/oracle bias becomes training bias and is not real feedback.
Illustrative example: Guessing a need can read better in turn one, while simulating another turn may show clarification prevents a wrong direction.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
What does adding more PTO branches expand compared with looking farther ahead?
Preference Tree Optimization: Enhancing Goal-Oriented Dialogue with Look-Ahead Simulations
Show answer and example
Answer: Width compares more actions at one state; depth observes consequences farther into the future. Both cost simulations and judgments but aren't substitutes. Shallow branches can miss delayed effects; one deep branch can miss better starts. This is a search distinction, not an empirical ranking.
Illustrative example: Five openings with one turn each increase width; one opening with many turns increases depth.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
Why preserve the mapping from current histories to future PTO branches?
Preference Tree Optimization: Enhancing Goal-Oriented Dialogue with Look-Ahead Simulations
Show answer and example
Answer: One preference pair compares actions from the same state. Continue each action in its own branch and do not leak future outcomes into policy inputs. Deterministic local evaluators replace the expensive oracle only to test tree and credit flow.
Illustrative example: If A prompts clarification and B confusion, attaching A's success to B reverses the preference.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Why does C²-DPO require a response pair and two contexts?
Show answer and example
Answer: Chosen/rejected compare responses to the same prompt; complete/degraded context varies evidence. They are independent axes. Hold the responses fixed to isolate context's effect.
Illustrative example: Compare red car and blue car with the original and degraded images, not with different questions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why compare full and degraded context margins rather than only raise chosen likelihood?
Show answer and example
Answer: Chosen likelihood may exploit language priors and ignore images. Context preference gain rewards the same pair changing preference with evidence while preserving the original ordering. It depends on appropriate degradation and does not itself establish correct visual understanding.
Illustrative example: If masking a chart leaves preference unchanged, language priors may dominate; correct evidence should provide extra support.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
Is positive context preference gain the same as the chosen response winning under full context?
Show answer and example
Answer: No. CPG compares the two margins; winning needs a positive full-context margin. An improved but negative margin can still favor the wrong reply. Context gain and original preference ordering are separate constraints.
Illustrative example: A change from −3 to −1 gives positive gain without a chosen win.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
What must stay consistent across C²-DPO's margins?
Show answer and example
Answer: Use one prompt and one chosen/rejected pair under both contexts. Degraded context must not contain the answer. Local work does not complete fine-tuning the named multimodal model/data.
Illustrative example: Subtracting a full-image A/B comparison from degraded-image C/D does not isolate visual evidence.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Why separate in-training and held-out performance in reward-model memorization?
What do Reward Models Memorize?
Show answer and example
Answer: A high score on a seen pair could reflect broad learning or memorization. Compare the same pair across actual runs including and excluding it; the difference measures conditional memorization.
Illustrative example: Compare models trained with this pair to models that did not see it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why sigmoid each reward margin before averaging by inclusion?
What do Reward Models Memorize?
Show answer and example
Answer: Sigmoid is nonlinear. Convert each actual run's margin into a probability, then separately average seen and held-out runs for memorization and generalization. Their difference can be negative. Group by actual manifests and keep users isolated.
Illustrative example: Large margin differences can be hidden by averaging before probability saturation; retain each run's membership.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
How does CM differ from training accuracy minus test accuracy?
What do Reward Models Memorize?
Show answer and example
Answer: CM compares the same pair across runs and splits by user to avoid leakage. Aggregate split means also mix difficulty differences. CM depends on split and training quality; it is not proof of verbatim memorization per item.
Illustrative example: A train–test gap between different user groups may reflect their difficulty rather than memory of any one preference.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
Can inclusion in reward-model training be inferred from a high score?
What do Reward Models Memorize?
Show answer and example
Answer: No. Use actual run manifests to assign inclusion and held-out status; scores are outcomes. Inferring inclusion from outcomes is circular. Check user-level isolation. The local aggregator does not create the paper's training runs.
Illustrative example: A high-scoring held-out pair stays held out; do not move it to support a memorization story.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Does SA-DPO measure semantic distance between responses or between prompts?
Mind the Gap: Structure-Aware Consistency in Preference Learning
Show answer and example
Answer: It compares preferred and dispreferred answers to one prompt. It is not a margin from prompt difficulty/length. Frozen response representations supply semantic distances independently of preference labels; distances aren't correctness scores.
Illustrative example: Two paraphrases of one answer can be close; two similar prompts are not the comparison unit.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why require less separation for semantically similar answers?
Mind the Gap: Structure-Aware Consistency in Preference Learning
Show answer and example
Answer: A large fixed margin spends capacity separating paraphrases and amplifies label noise. SA-DPO scales the added margin with frozen semantic distance, zero at zero distance, while keeping relative-to-reference preference. Control encoder quality and mean pressure separately.
Illustrative example: 'The answer is three' and 'The result is 3' need less separation than right vs wrong answers; wording can still bias distances.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
Why match mean margin when comparing SA-DPO to a fixed-margin method?
Mind the Gap: Structure-Aware Consistency in Preference Learning
Show answer and example
Answer: Otherwise performance can change from overall training pressure rather than semantic allocation. Matching means better isolates allocation without proving semantic judgments always correct.
Illustrative example: An easier objective from a smaller mean margin does not establish better semantics.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
What should SA-DPO do when response distance is zero?
Mind the Gap: Structure-Aware Consistency in Preference Learning
Show answer and example
Answer: Set extra distance margin to zero, but retain the policy/reference log-ratio difference in preference loss. Use frozen distances, not constants or test-label calibration.
Illustrative example: Removing extra separation for identical wording does not make the whole loss zero.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Does GroupDPO concatenate all responses into one long response?
GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization
Show answer and example
Answer: No. One prompt has nonempty positive and negative sets; responses retain separate probabilities and IDs. The joint objective uses set relationships without requiring simultaneous activation storage.
Illustrative example: Two positive and two negative answers are encoded independently while the objective compares both sets.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why not recompute GroupDPO coefficients inside each response microbatch?
GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization
Show answer and example
Answer: Freeze derivatives from the entire group at the same parameter point and replay response gradients to recover the first-order result. Local recomputation changes sets and normalization. Recompute after every update. Loss values and Hessians are not equivalent.
Illustrative example: Each response carries fixed full-group credit rather than re-ranking within a small compute batch.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
Are GroupDPO Margin, MPO, Softmax and All-Pairs equivalent objectives?
GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization
Show answer and example
Answer: No. They compare sets differently. What is shared is precomputing derivatives of the chosen full-group objective and replaying separately. Changing the objective differs from changing compute batches while retaining its first derivative.
Illustrative example: All pairwise comparisons may assign different coefficients than pooling positives first; the same memory method serves both.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
Can GroupDPO coefficients be reused after the optimizer step?
GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization
Show answer and example
Answer: No. Derivatives change with parameters. Recompute. Do not divide microbatch backpropagation by local groups or microbatch count; positive NLL also uses the full-batch group normalization.
Illustrative example: Reusing old coefficients saves a pass but becomes a stale approximation, not exact current gradients.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Why should a probabilistic autorater output a preference probability, not just a winner?
Judging with Confidence: Calibrating Autoraters to Preference Distributions
Show answer and example
Answer: Real preferences can disagree. Brier/log rewards fit observed preference frequencies rather than force binary certainty. Reliable target distributions matter; arbitrary numbers are not calibrated.
Illustrative example: Near-even annotator votes warrant an intermediate probability rather than an absolute winner.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why can't Autorater backpropagate through parsed probability text?
Judging with Confidence: Calibrating Autoraters to Preference Distributions
Show answer and example
Answer: The policy generates text, parses it to a fixed reward and uses policy gradients. Discrete text-to-float parsing has no ordinary differentiable path. Brier/log rewards encourage calibration, with separate invalid penalties. Derivatives of numeric scores alone are not GRPO.
Illustrative example: An unformatted 'very confident' response fails parsing; don't silently substitute a plausible number.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
How do Brier and log rewards penalize confident errors differently?
Judging with Confidence: Calibrating Autoraters to Preference Distributions
Show answer and example
Answer: Both are proper probability scores, but squared and logarithmic tails differ; log loss is especially sensitive to extremely confident mistakes. Parse failures also have a separate reward, beyond legal probability gradients.
Illustrative example: A near-certain win prediction followed by defeat is penalized by both, with different magnitudes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
Why can't invalid numbers enter the formula before multiplying by a zero mask?
Judging with Confidence: Calibrating Autoraters to Preference Distributions
Show answer and example
Answer: Separate invalid outputs first. NaN can propagate through nonlinearities and multiplication by zero; out-of-range values are not probabilities. Use the parse-failure reward. The local strict parser is a contract, not the authors' parser or full RL.
Illustrative example: Route NaN directly to invalid reward rather than computing log(NaN) first.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Which axis is target, which is benchmark role, and which cell measures self-evaluation?
When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation
Show answer and example
Answer: The target is evaluated; the benchmark role performs evaluation. Self-evaluation is the diagonal where they match. Compare own and other roles for one fixed target, not different targets.
Illustrative example: A answering with B judging is off-diagonal; A answering and judging is diagonal. Keep A as the target in both.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why rank each segment and exclude self from the peer mean?
When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation
Show answer and example
Answer: Segment scales may differ, so rank before averaging to preserve the protocol. Remove self from the peer comparison. Score direction, axes and tie handling affect the sign; peer consensus is not external truth.
Illustrative example: A low-is-better metric and high-is-better metric need consistent rank direction. Never put the self score back into peers.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
How does the protocol distinguish self-evaluation advantage from a stronger model?
When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation
Show answer and example
Answer: Compare one target's ranks under its own role and peers to isolate evaluator preference from target quality. Peers can share biases, so this is relative audit evidence, not a definitive capability verdict.
Illustrative example: Consistently leading under all judges differs from leading only under one's own judge.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
Why do axis order and leave-self-out matter for self-bias?
When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation
Show answer and example
Answer: Rank within segments, average over segments, then subtract the peer mean excluding self from the diagonal. Specify whether scores are higher- or lower-is-better; negative bias indicates favorable self-rank.
Illustrative example: MetricX-QE is lower-is-better. Descending ranks would reverse the bias direction.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Does reference-free SimPO mean there is no training feedback?
Show answer and example
Answer: No. Chosen/rejected pairs still supply preferences. It removes a resident reference model, using length-averaged policy log-probability as implicit reward with a fixed margin.
Illustrative example: You still need a better answer label; no frozen reference scores need be computed.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Without a reference, how does SimPO avoid favoring short responses by sequence probability?
Show answer and example
Answer: Total log probability accumulates per token and changes reward scale with length. Average per token and require the preferred response to beat the rejected one by a margin, without an extra reference forward. This removes one length effect, not all quality bias.
Illustrative example: A short wrong answer can beat a longer correct one on summed log probability; averaging removes the direct sum-length penalty.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
Is SimPO's training margin equivalent to reducing inference temperature?
Show answer and example
Answer: No. Margin sets desired average-reward separation and changes parameter gradients. Temperature rescales next-token logits at inference. They can coexist but shorter output at one temperature does not validate the margin.
Illustrative example: Training compares complete response average log probabilities; inference temperature doesn't read that preference pair.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
Which tokens enter SimPO's log-probability sum and length?
Show answer and example
Answer: Use the same valid-response mask for both; no padding or prompt tokens. The loss updates only the policy. A token objective does not reproduce the paper's judge/model matrix.
Illustrative example: Counting padding in the denominator distorts the short answer's average reward.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
What is the difference between ORPO odds and probability?
Show answer and example
Answer: Probability p is an event's share of all outcomes. Odds p/(1−p) compare occurrence to nonoccurrence and can exceed one. ORPO compares preferred and rejected odds, not percentages.
Illustrative example: p=1/2 gives odds 1; p=3/4 gives odds 3, not 300% probability.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
What do SFT and the odds-ratio term each constrain in ORPO?
Show answer and example
Answer: SFT raises chosen likelihood; the odds term widens separation from rejection. Both update one policy without reference in the objective. Odds are boundary-sensitive. Reporting reference KL as a diagnostic does not add it to training.
Illustrative example: Raising only chosen likelihood can also raise similar bad answers; odds explicitly separate them.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
How does ORPO's single stage differ from SFT followed by separate preference optimization?
Show answer and example
Answer: One policy update combines chosen NLL and preference odds without a resident frozen reference for another stage. Two-stage training first produces an SFT policy then changes objective/reference. One stage is not universally better.
Illustrative example: One preference batch supplies NLL and odds gradients instead of finishing chosen-only SFT first.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
Why handle probabilities near zero or one in ORPO's log-odds margin?
Show answer and example
Answer: Log odds are log(p)−log(1−p), then contrasted across replies. Boundaries cause log zero or huge ratios. The local kernel adds 1e−12 to both terms and uses logaddexp, not probability subtraction. Candidate probabilities are not token-level response probabilities.
Illustrative example: 0.9→0.99 is +0.09 probability but odds go 9→99.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Are KTO desirable and undesirable weights feedback probabilities?
Show answer and example
Answer: No. Labels describe single accepted/rejected outputs; λ weights control class utility contributions. Class frequency and explicit weight are separate, neither a predicted satisfaction probability.
Illustrative example: Rebalance many upvotes and few downvotes without inventing matched same-question pairs.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
How does KTO update from single desirable feedback without same-prompt pairs?
Show answer and example
Answer: Use response policy/reference log ratio against expected KL as a dynamic baseline and define separate desirable/undesirable utilities. Independent examples can be reweighted without fictitious comparisons. Reference and imbalance still matter; a label is not absolute reward.
Illustrative example: Upvote and downvote on separate inquiries enter separate branches, not a fabricated preference pair.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
Does KTO differ from DPO only by unpairing the data?
Show answer and example
Answer: No. KTO defines gains/losses around a KL reference point; DPO compares same-prompt response margins. Standalone feedback cannot be arbitrarily paired, and class weights alter tradeoffs.
Illustrative example: One user's upvote and another's downvote on different queries do not order their answers to one prompt.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
What is KTO's reference point, and what does the local policy establish?
Show answer and example
Answer: It is expected policy-to-reference KL, not one answer's log ratio. Separate aggregate baseline and sample signal. The local six-candidate linear policy checks this relation, not free generation.
Illustrative example: Compare a high log ratio to the moving baseline; don't treat the example as the entire KL expectation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Does a Self-Rewarding judge tie reveal the preferred answer?
Show answer and example
Answer: No. A tie gives no strict preference; display order cannot create one. Candidate count and distinguishable judging are separate. Even unequal scores are the current judge's judgment, not new human truth.
Illustrative example: If two candidates tie, don't label the first shown the winner.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Where do Self-Rewarding LM preferences come from, and why can errors compound?
Show answer and example
Answer: Each round ranks its own generated candidates and trains the next round with DPO on those judgments. They are not independent human labels; persistent judge biases can become stronger. Keep external held-out evaluation.
Illustrative example: A judge rewarding verbosity may repeatedly choose long arithmetic errors; an external check can break the loop.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
How does Self-Rewarding differ from a fixed external reward model?
Show answer and example
Answer: A fixed judge doesn't coevolve; self-reward hopes both generation and judgment improve but can amplify shared errors. Compare at equal budget with independent evaluation, not rising self-scores.
Illustrative example: A model can learn to please its own scoring rule without becoming more useful to users.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
What provenance must Self-Rewarding iterations retain?
Show answer and example
Answer: Record generating policy round, judge round and pair-selection rule. Don't backfill old data with post-update scores. Use isolated evaluation. Local tests cover state and objective, not paper judge quality.
Illustrative example: After training a new model, don't rescore old candidates and present those scores as original supervision.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Where do SPIN positives come from without new human preference data?
Show answer and example
Answer: Existing human SFT demonstrations stay positive. The previous policy generates negatives on the same prompts. Self-play changes negatives and reference; it still uses demonstrations and generated replies are not necessarily factually wrong.
Illustrative example: A standard answer already exists; generate a comparison without asking a human to rank it again.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
How does SPIN's previous-policy negative differ from self-judged preferences?
Show answer and example
Answer: Human demonstrations are positives; prior-policy samples are negatives. Train relative to the prior policy, then refresh it. No self-judge scoring is needed. Human supervision remains and negatives aren't necessarily false.
Illustrative example: Use fixed expert examples and regenerate opponents next round instead of replaying all old negatives.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
How does SPIN's refreshed negatives differ from repeated SFT on fixed examples?
Show answer and example
Answer: SFT raises fixed demonstration likelihood. SPIN resamples a comparison distribution from the previous policy on the same prompts and distinguishes demonstrations from it. Both negatives and relative reference change, not just training rounds.
Illustrative example: After fixing earlier mistakes, the next opponent may make different ones. Fixed SFT won't refresh those negatives.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
Why freeze SPIN's opponent during a round and refresh it afterward?
Show answer and example
Answer: The previous policy supplies both samples and the reference. Changing it mid-round changes the objective. Refresh after the round to align generation, scoring and updates. No full-scale local text experiment.
Illustrative example: Using old reference for half the batches and a new one for the rest does not define one consistent round.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Does increasing τ increase or decrease IPO's target gap?
Show answer and example
Answer: The finite target is 1/(2τ), so increasing positive τ decreases the gap. It is not a label-strength score; training outcomes also depend on data, policy and optimization.
Illustrative example: Changing τ from 1 to 2 changes 1/2 to 1/4; this is formula arithmetic.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why fit a finite IPO gap rather than always separating chosen and rejected farther?
Show answer and example
Answer: Logistic losses can keep enlarging a separable margin, risking overfit. IPO squares error to a finite reference-relative log-ratio target. Overshooting reverses the gradient. This changes loss shape, not just learning rate.
Illustrative example: Once separation is sufficient, continued suppression of rejection need not help; excess separation is corrected.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
Does fitting IPO's gap require the chosen response's absolute probability to increase?
Show answer and example
Answer: No. The gap is chosen-minus-rejected policy/reference log ratios. Raising chosen, lowering rejected or both can reach the target. Chosen probability alone omits the relative comparison and global normalization.
Illustrative example: Chosen can stay fixed while rejected falls and the gap grows.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
Why use a cross policy/reference probability ratio in IPO?
Show answer and example
Answer: It measures how much the preferred-over-rejected relationship changed relative to reference, not just policy probability difference. Use a fixed positive τ and target 1/(2τ), aggregate response tokens correctly and retain reference terms.
Illustrative example: If reference already favors a wording, the cross-ratio distinguishes inherited preference from added change.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Are SteerLM's attribute predictor and conditioned generator the same role?
Show answer and example
Answer: No. The predictor annotates existing answers with helpfulness, quality and other dimensions. The generator learns to produce answers under target attributes supplied at inference; it does not need a future answer scored first.
Illustrative example: Annotate training replies as concise/helpful, then request new replies conditioned on those attributes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why condition generation on attributes instead of aggregating all rewards?
Show answer and example
Answer: A single scalar makes preference mixtures hard to change at inference. Predict attributes, then train conditional SFT so users can specify combinations. Labels must be reliable, and control does not guarantee precise independent satisfaction.
Illustrative example: Ask for detailed but restrained or concise and friendly without retraining a policy for each combination.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
How does SteerLM differ from asking for brevity in an inference prompt?
Show answer and example
Answer: Attribute-conditioned SFT systematically associates labeled values with response patterns. A one-off natural-language instruction needn't have that training signal. They can coexist, but adding attribute text alone is not conditioned training.
Illustrative example: Training examples across concision levels give an inference level learned meaning beyond wording.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
Are local SteerLM attribute labels equivalent to the authors' predictor?
Show answer and example
Answer: No. Deterministic reward axes on public candidates supply local labels to test the interface. No large attribute predictor or free-text conditioned generator is trained. Preserve attribute order and value domains.
Illustrative example: Swapping quality and length columns may run while reversing user controls.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Which human work does RLAIF replace?
Show answer and example
Answer: AI compares candidate preferences to reduce pairwise human labeling and supplies strategy training signals. Evaluation criteria and independent assessment remain. AI labels can have position, style and knowledge biases.
Illustrative example: People can specify criteria while AI compares summaries; its scores aren't objective truth.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why swap candidate order for RLAIF judging, and can raw scores be averaged?
Show answer and example
Answer: Judges may favor earlier positions. Evaluate both orders and combine only after mapping to the same candidate identity. Raw positional averaging can cancel a true preference. It controls position, not factual, style or value biases.
Illustrative example: Convert both trials to A's win score before averaging.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
How does RLAIF training differ from an AI choosing an answer at inference?
Show answer and example
Answer: Training converts judgments to data and updates parameters. Inference selection changes one response without learning. Similar judges may be used, but costs and effects on future requests differ.
Illustrative example: Selecting a reply today changes today's output; training on the pair may change tomorrow's policy.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
How does local RLAIF keep identities straight when swapping order?
Show answer and example
Answer: Map each judgment back to original candidate IDs before combining labels. Fixed labels verify mapping and update, not a full judge or large RL.
Illustrative example: If left wins on pass two and B is now left, credit B rather than A.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Why doesn't DPO need a separately fitted reward model?
Show answer and example
Answer: The KL-regularized policy expresses implicit reward as a policy/reference log ratio. Direct paired classification uses chosen and rejected labels. Reward information still comes from preference supervision.
Illustrative example: With labels on two answers, compute the paired loss without first training a scorer.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why use DPO policy changes relative to reference instead of raw answer probabilities?
Show answer and example
Answer: Raw scores inherit pretraining preferences. Relative ratios measure redistribution against the baseline, jointly comparing preferred/rejected in the classification margin implied by KL regularization. No explicit reward model, but same-prompt preferences and fixed reference remain.
Illustrative example: If a phrase starts with high likelihood, distinguish inherited preference from a further supervised increase.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
How does DPO's data flow differ from reward-model PPO?
Show answer and example
Answer: Classic DPO consumes fixed preference pairs without sampling and reward-model scoring at every step. PPO usually includes those stages. A fixed dataset may omit states future policies visit.
Illustrative example: A preference bank without a wrong-prefix category doesn't automatically gain it from offline updates.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
Why freeze DPO's reference and align response masks?
Show answer and example
Answer: The reference anchors changes; updating it changes the objective. Score identical response tokens in both paths, with neither prompt nor padding mismatches.
Illustrative example: Adding prompt likelihood only to reference introduces irrelevant input length into the margin.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Why combine sequence calibration and supervised loss in SLiC-HF?
Show answer and example
Answer: Ranking makes preferred likelihood exceed rejected likelihood. Supervision learns trusted content rather than succeeding only by suppressing bad replies. Both train the generator directly.
Illustrative example: Separate good from bad summaries while learning the good summary itself.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why keep supervised cross-entropy when SLiC-HF already has a preference margin?
Show answer and example
Answer: Relative ranking does not prevent overall language drift. Cross-entropy preserves reference response likelihood, combining calibration and generation learning. Off-policy data do not remove coverage, length or source differences.
Illustrative example: Suppressing bad answers can satisfy ranking even while good answers get worse; supervision retains good generation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
How does SLiC-HF's hinge margin differ from DPO's reference-relative one?
Show answer and example
Answer: The hinge directly constrains sequence likelihood differences alongside supervision. DPO contrasts log ratios relative to a frozen policy. Both baseline and loss shape differ.
Illustrative example: Once the hinge margin is met, ranking penalty can stop; reference-relative preference is different.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
Why can SLiC-HF use pairs collected by another model?
Show answer and example
Answer: The objective can score fixed responses and doesn't need current-policy samples. Keep sources and same-prompt pairing. Local candidate calculations are not the paper's T5 and human evaluation.
Illustrative example: Old summaries can be scored by the current policy if labels still belong to those summaries.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
Why use a full ordering of multiple same-prompt answers in RRHF?
Show answer and example
Answer: Multi-source responses provide more relative comparisons than one pair. Match length-averaged log likelihood to reward order and supervise the best answer. Ranking quality depends on labels/reward.
Illustrative example: Good, middling and poor responses supply several orderings, not one winner pair.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why still apply SFT to the best answer after ranking responses?
Show answer and example
Answer: The hinge only fixes ordering conflicts and can stop giving a useful generation signal once sorted. SFT raises high-quality content likelihood. External rewards and length normalization do not guarantee factual correctness.
Illustrative example: Even with three answers ranked correctly, the best may be rare; its SFT term can still increase likelihood.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
What do best-response SFT and RRHF ranking each add?
Show answer and example
Answer: Ranking corrects relative order; SFT directly raises best-response likelihood. Ranking alone can succeed by suppressing alternatives. The combination isn't a universal optimum.
Illustrative example: A good answer should be learned, not merely score above a bad one.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
Why isn't RRHF p_i raw probability?
Show answer and example
Answer: It is length-normalized response log probability. The hinge compares this order within one prompt with consistent reward direction. Candidate tests do not reproduce full 7B training.
Illustrative example: Using decimal probabilities in place of mean logp changes margins and gradients.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Basics
What role does the constitution play in Constitutional AI?
Show answer and example
Answer: Explicit natural-language principles guide critique, revision and AI preference judgments. They supply human supervision, not absolute values discovered by the model.
Illustrative example: Use a stated principle to explain a revision instead of an unexplained good/bad score.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Core mechanism
Why are critique/revision and AI preference training separate stages?
Show answer and example
Answer: First critique and revise by principles, then supervise the revised texts. Next compare responses by those principles and use preference learning for RL. Demonstrations and choice labels differ. Principles and local updates do not establish harmlessness or full red teaming.
Illustrative example: Rewrite a reply to avoid privacy leakage, then compare two rewrites; text and winner labels are distinct inputs.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Method comparison
Why distinguish principle-based revision SFT from RLAIF?
Show answer and example
Answer: The former learns revised text; the latter uses AI preferences to train reward and optimize policy. Data and execution differ. Local simplification does not establish a separate preference model or PPO; arithmetic principles are not safety red-team evidence.
Illustrative example: Repairing one answer supplies a demonstration, not evidence of completed reward-driven RL.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
LLM post-training / Preference and alignment: identify the feedback first
Implementation details
What safety claims do local Constitutional AI principles support?
Show answer and example
Answer: Deterministic axes such as correctness, format, process and brevity test principle scoring and state updates. They do not cover real harmful requests or train a complete safety reward model.
Illustrative example: Valid formatting doesn't establish reliable refusal of dangerous advice.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Topic guide and reading order
Start with this question¶
How do we turn pairwise preferences or a thumbs-up on one answer into a trainable objective?
Takeaway¶
DPO uses preferred and rejected answers to the same prompt and classifies preferences through the current-to-reference policy log ratio. KTO uses individually desirable or undesirable feedback without pairs. Reward models, AI raters, and controllable SFT change the source or organization of feedback. The feedback format, reference distribution, and annotation reliability constrain the objective; algorithm names alone do not justify swapping losses.
Example¶
A support system may have ranked pairs for one group and accept/reject labels on individual answers for another. Use true same-prompt pairs for DPO, not arbitrary pairs from different questions. If groups disagree on style, retain preference distributions and annotation provenance so a judge's style preference is not mistaken for factual correctness.
Common pitfall¶
Better preference scores do not guarantee facts, safety, or production gains. Match data, reference model, and evaluation, and check length/style confounds. Closed-form losses, candidate policies, and preference fixtures validate their own mechanisms. A scorer contract or an unreproduced experiment does not show the model is aligned.
Reading order¶
Read DPO for paired feedback and the reference log ratio, then KTO for unpaired binary feedback, then probabilistic Autorater for confidence distributions rather than a single winner.
- DPO · Original paper
- KTO · Original paper
- Judging with Confidence: Calibrating Autoraters to Preference Distributions · Original paper
Check your understanding¶
Can a liked answer and a disliked answer to a different question form a DPO pair?
Show suggested answer
Usually not. Standard DPO compares answers under the same condition. Cross-question pairing confounds difficulty and content. Collect same-prompt alternatives or use an unpaired objective with explicit label distribution, reference policy, and independent evaluation.