Preference is not one score: establish feedback format before choosing an alignment objective¶

Updated 2026-10-27 · 25 topic entries / 6 representative comparisons

DPO, KTO, SimPO, ORPO, and IPO change how feedback is used; CleanRM first asks whether feedback is reliable. References, length normalization, finite margins, and label-noise corrections are independent axes even if all distinguish better from worse responses.

Mechanisms · Trade-offs · Cross-paper questions · Updates

How to read this card

The table compares representative methods. It is not a shared-benchmark leaderboard. Teaching examples are not measured results.

Mechanism comparison¶

Axes: Supervision unit: same-prompt pairs, individual desirability labels, or noisy labels · Reference and scale: reference log-ratio, length average, and odds · Objective shape: increasing separation, finite margin, or joint SFT and preference · Source of error: excessive optimization pressure versus noisy labels

Paper and source	Mechanism	Benefits and trade-offs
DPO · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Classifies chosen versus rejected answers to the same prompt using log-ratios against a frozen reference, without training a separate reward model.	Removes explicit reward-model training and online RL but still requires reliable preference pairs. The local candidate-action loss does not train the original autoregressive LLM.
KTO · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Uses individual desirable/undesirable feedback and a KL reference point from policy to reference to construct utility.	No manufactured same-prompt pairs are needed. Class weights, imbalance, and reference distribution still matter. The local implementation uses a six-candidate linear policy.
SimPO · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Uses per-token mean policy log-probability as implicit reward with a fixed preference margin, omitting reference forwards.	Length scale and preference pressure are explicit. No reference does not mean no feedback. A real token-loss path exists, but original model scale and judge were not reproduced.
ORPO · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Combines chosen-answer SFT NLL with a chosen/rejected odds-ratio preference term.	One stage trains absolute likelihood and relative preference, but odds have numerical boundaries. Local complete-candidate actions do not reproduce original LM training.
IPO · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Fits a finite reference-relative preference gap of 1/(2τ) rather than requiring indefinite separation.	Overshooting also draws the gap back, changing objective shape. A reduced token-level free-generation path exists, but cannot establish original large-model performance.
CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 结果、代码映射与边界	Uses clean calibration data and joint optimal transport over semantics and recovered preferences to estimate two-way noise, then freezes the estimator to correct reward-model risk.	Depends on noise rates and calibration representativeness; do not clip signed losses to zero. Full RM/RLHF was too costly and not reproduced; local evidence is an L1 component.

Trade-off analysis¶

Start from observed data, not labels implied by a fashionable loss¶

DPO, SimPO, ORPO, and IPO use relative outcomes between responses to the same question. KTO can use single-answer feedback. Pairing reactions across questions invents an unobserved comparison. CleanRM instead estimates how noisy labels differ from clean preferences and corrects training risk; swapping the preference objective does not do this.

Sources: DPO · KTO · CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF

Reference-free and less overfitting are separate choices¶

SimPO removes the reference but keeps mean token reward and a fixed margin. ORPO combines SFT and odds preference. IPO keeps the reference and fits a finite gap. Analyze memory, length scaling, chosen-text likelihood, and saturation separately rather than collapsing them to one local accuracy number.

Sources: SimPO · ORPO · IPO

Cross-paper questions¶

Try answering before expanding the explanation. Each question relates at least two papers.

Should individual thumbs-up/down feedback be forced into DPO pairs or handled with KTO?¶

Show answer and example

Answer: Preserve the observed feedback unit. DPO needs a same-prompt comparison; KTO uses individual desirability and a KL reference. Reactions across prompts do not form a preference pair. Collect or reliably construct real same-prompt comparisons before choosing a pairwise loss.

Illustrative example: Teaching example: A thumbs-up on question A and thumbs-down on question B do not say which two answers to A are preferred.

Sources: DPO · KTO

Can SimPO and ORPO be interchanged by turning off DPO's reference?¶

Show answer and example

Answer: No. SimPO compares length-averaged policy log-probabilities against a fixed margin. ORPO also improves chosen likelihood and compares odds. Removing a reference does not supply either definition or equalize length and probability bounds.

Illustrative example: Teaching example: ORPO's chosen NLL independently contributes gradients even when preference separation increases; it is not a SimPO margin.

Sources: SimPO · ORPO · DPO

Can a finite IPO margin fix a correct answer mislabeled rejected?¶

Show answer and example

Answer: Not automatically. IPO bounds the target gap but still uses the supplied direction. CleanRM instead estimates noise and corrects risk, contingent on estimator quality. Reduced pressure does not clean labels.

Illustrative example: Teaching example: Making the actually better answer lose by a smaller amount still does not reverse the wrong label.

Sources: DPO · IPO · CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF

Update log¶

Updates record new approaches, changed trade-offs, and follow-up evidence.

2026-10-27 · Initial edition: compares DPO/KTO feedback units, SimPO/ORPO reference-free objectives, IPO finite margins, and CleanRM noise correction; 25 abstracts reviewed and six papers read closely.

Covers 25 topic entries; see the full list below.

All 25 topic entries

The full catalog follows. The main comparison covers the representative papers listed above.

Paper-by-paper questions · All deep dives