跳转至

LLM post-training by year

简体中文 · English

Browse by first public release year. Papers are listed newest first within each year with a short method description.

2026

2025

  • 2025-12 · Online IcePop (online-icepop): Online IcePop allows only one update per batch, so the stale-policy ratio is 1 and PPO ratio/clipping are removed. Keep the IcePop two-sided mask and original in-range engine-mismatch ratio.
  • 2025-12 · Stabilizing RL with LLMs (minirl): Separate train–inference mismatch from policy staleness. Use on-policy importance correction and combine off-policy clipping with MoE Routing Replay.
  • 2025-10 · IcePop (icepop): One-sided TIS may retain very small mismatched ratios. Mask ratios outside a fixed two-sided interval and retain original corrections inside.
  • 2025-09 · Judging with Confidence: Calibrating Autoraters to Preference Distributions (probabilistic-autorater): The strict local tag parser is an explicit local choice, not the unpublished author parser. A numerical gradient diagnostic does not establish backpropagation through text-to-float parsing or GRPO training. Bounded component only; full experiments are not reproduced.
  • 2025-09 · When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation (benchmark-self-bias): Average ranks for ties are an explicit local choice. Peer consensus is not absolute truth, and synthetic tables do not establish real-model bias. Bounded component only; full experiments are not reproduced.
  • 2025-08 · CHORD (chord): Separate SFT and RL stages can overfit expert data or forget it too early. Add expert SFT as an annealed auxiliary term during on-policy RL; token-level uncertainty weights smooth the transition from imitation to exploration.
  • 2025-08 · GPPO (gppo): This can suppress exploration and learning from negative samples. GPPO keeps PPO's forward clipped objective while restoring backward signals in those regions with a stop-gradient boundary weight.
  • 2025-08 · TIS (tis): TIS multiplies policy gradients by the training-to-rollout probability ratio. A one-sided upper cap controls heavy-tail variance while retaining small-weight samples.
  • 2025-07 · GSPO (gspo): GSPO averages the log-ratios for a response, then exponentiates the result to get one sequence ratio. The entire sequence shares the clipping weight.
  • 2025-07 · Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (rlcr): RLCR predicts both an answer and its confidence and adds a bounded proper scoring rule so honest probabilities maximize expected reward. The local baseline minimizes multiclass Brier over the full candidate distribution.
  • 2025-06 · SPIRAL (spiral): The same model plays problem setter and solver. Automatically scored zero-sum multiround games create a progressively harder curriculum.
  • 2025-06 · CISPO / MiniMax-M1 (cispo): Sample with a fixed rollout policy and compute token-level importance ratios. Clipping only the ratios preserves the direction of advantages and effective gradients.
  • 2025-05 · INTUITOR (intuitor): The direction is uniform-to-policy, not negative policy entropy. Confidence does not imply correctness.
  • 2025-05 · SCAR: Shapley Credit Assignment for More Efficient RLHF (scar-shapley-credit): The local scope is exact enumeration for N<=10 and reward scatter at gamma=1. It does not treat an unspecified parser, an Owen hierarchy, or a synthetic reward model as the original system. Bounded component only; full experiments are not reproduced.
  • 2025-05 · Absolute Zero (absolute-zero): A proposer generates verifiable tasks, a solver attempts them, and an executable verifier supplies rewards. Adapt task difficulty to the current capability boundary.
  • 2025-04 · TTRL (ttrl): Sample several answers for the same test question, use the majority answer as a pseudo-label, and update immediately without gold labels.
  • 2025-04 · LUFFY (luffy): Put high-quality offline reasoning and online rollouts on the same support. Regularize importance ratios to retain on-policy behavior.
  • 2025-04 · VAPO (vapo): Critic bias, heterogeneous lengths and sparse rewards destabilize value-based PPO on long traces. VAPO pretrains a value model and adapts GAE and actor updates to response length.
  • 2025-03 · Dr. GRPO (dr-grpo): GRPO's response-length averaging and within-group standard deviation can introduce length and problem-difficulty biases. Remove both normalizations while keeping centered group-relative reward so trajectories retain their scale.
  • 2025-03 · DAPO (dapo): Standard symmetric clipping can suppress exploration in online RL for long chains of thought, and groups with all correct or all wrong answers offer no learning signal. DAPO's Clip-Higher gives more room to increase probabilities; Dynamic Sampling discards zero-variance groups; token-level loss handles responses of different lengths fairly; and overlong responses receive a piecewise penalty.
  • 2025-01 · REINFORCE++ (reinforce-plus): Prompt-local standard deviations in GRPO/RLOO can randomly reweight groups of different difficulty. REINFORCE++ retains within-group centering but normalizes by a global advantage scale across batches to reduce variance and local bias without a critic.

2024

  • 2024-05 · SimPO (simpo): DPO keeps a reference model and sequence probability tends to favor short responses. SimPO uses mean token log-probability as the implicit reward, drops the reference, and adds a fixed margin to Bradley–Terry preference training.
  • 2024-03 · ORPO (orpo): ORPO combines the chosen response's NLL and a chosen-versus-rejected odds-ratio penalty into one objective. Near probabilities of 0 or 1, odds provide a more sensitive contrastive signal than raw probability differences.
  • 2024-02 · RLOO (rloo): RLOO treats a full response as one action. Sample several responses for the same prompt and use the other responses' mean reward as the current one’s baseline.
  • 2024-02 · DeepSeekMath / GRPO (grpo): GRPO samples a group of responses to each question and computes advantages from the group's reward mean and standard deviation, removing the critic. The policy still uses an old-policy ratio, clipping, and reference KL.
  • 2024-02 · KTO (kto): It uses the expected KL between policy and reference as a reference point. Examples of each class can be collected independently and need not be balanced.
  • 2024-01 · Self-Rewarding LM (self-rewarding): The current model generates candidates and judges them each round. New preference pairs feed the next DPO step, forming a bootstrapping loop.
  • 2024-01 · SPIN (spin): Starting from an SFT model, SPIN uses the previous model to answer training prompts, treats human demonstrations as positives and self-generated answers as negatives, and trains the next model with a self-play discriminative objective. This repeats without new human preference labels.

2023

  • 2023-12 · Math-Shepherd (math-shepherd): Sample several continuations from an intermediate reasoning step. Their final answer correctness supplies automatic step labels for verifier training and reranking.
  • 2023-10 · IPO (ipo): The paper proposes \(\Psi\)PO, which optimizes preference probabilities directly. The identity mapping yields IPO: fit a finite target margin instead of indefinitely widening the chosen/rejected gap with logistic loss on separable training data.
  • 2023-10 · ReMax (remax): LLM rollouts have fast simulation, deterministic token transitions and typically terminal rewards. Drop PPO's value model and subtract the current policy's greedy reward as a prompt-dependent baseline for lower-variance REINFORCE.
  • 2023-10 · SteerLM (steerlm): Traditional RLHF reduces multidimensional preferences to one reward, and users cannot change the target at inference. SteerLM uses an attribute predictor to label responses for helpfulness, quality, and other properties, adds attributes and values as conditions in ordinary SFT, and lets the user specify target attributes at inference.
  • 2023-09 · RLAIF (rlaif): An AI labeler judges candidates in both presentation orders. Average away positional bias and train the preference policy with the paired labels.
  • 2023-06 · GKD (gkd): GKD lets the student generate rollouts from its current policy and queries the teacher for full distributions at the states the student actually visits. The student data fraction interpolates fixed and on-policy data, and the divergence may be forward KL, reverse KL, or generalized JSD.
  • 2023-06 · MiniLLM (minillm): Forward KL covers all teacher probability mass, so a small student may overestimate low-probability teacher regions. MiniLLM optimizes mode-seeking reverse KL on the student's own generation distribution and stabilizes policy gradients with teacher-mixed sampling, single-step decomposition, length normalization and a reward baseline.
  • 2023-05 · Let's Verify Step by Step (process-supervision): A step-level reward model judges each reasoning step and prioritizes uncertain steps for annotation. The local comparison uses the same candidates and budget as outcome-only reward.
  • 2023-05 · DPO (dpo): Conventional RLHF fits a reward model and then optimizes the policy with PPO, making training complex and unstable. DPO starts from the optimum of KL-regularized RLHF, writes the implicit reward as a policy-to-reference log ratio, and reduces training to binary classification on preference pairs.
  • 2023-05 · SLiC-HF (slic-hf): Instead of a multi-model PPO-RLHF stack, enforce a margin between preferred and rejected sequence log-likelihoods while regularizing with SFT. The objective can use off-policy preferences collected for other models.
  • 2023-04 · RAFT (raft): RAFT samples multiple responses from the current model each round, ranks them with a reward model, discards lower-quality samples and maximum-likelihood fine-tunes on the selected responses. The updated policy starts the next round.
  • 2023-04 · RRHF (rrhf): PPO-RLHF coordinates policy, old policy, reward and value models, making training and tuning complex. RRHF samples responses from several models or humans, uses reward to rank them all, aligns the model's average log-likelihood order with reward order, and applies SFT to the highest-quality answer.

2022

  • 2022-12 · Constitutional AI (constitutional-ai): In stage one, the model critiques and rewrites its responses according to the principles, then learns by SFT on the revisions. In stage two, AI compares responses, a preference model is trained, and its reward drives RLAIF.
  • 2022-03 · InstructGPT / PPO-RLHF (ppo-rlhf): Increasing language-model size alone does not guarantee alignment with user intent. InstructGPT establishes the three-stage pipeline: SFT on annotator demonstrations, reward-model training on pairwise rankings, and PPO policy optimization with a KL penalty to limit drift from the SFT/reference model.