跳转至

LLM post-training by topic

简体中文 · English

Browse two levels: research area, then comparable method family. Each paper has its own entry. Open its write-up for experimental results and reproduction limits.

Rewards, credit, and curricula

Curricula and capability boundaries

  • Learning to Reason with Selective Internal Feedback (sif): It first votes over windows, then derives a binary reward from the fraction of eligible peers beaten. Ineligible responses and constant-reward groups are filtered; policy and KL terms both use the selection mask. Bounded component only; full experiments are not reproduced.
  • A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPO (grpo-difficulty-curriculum): Advancing to B5 can hurt results. This implementation reestimates difficulty from actual rollouts at every step, retains B5 as a control, and does not substitute problem length or stale scores.
  • BoundaryEvo: Data-Model Co-Evolution at the Capability Boundary for Long-Chain Reasoning Distillation (boundary-evo): Replay questions that the student has not yet mastered are retained, and new questions are added. Fresh teacher trajectories are generated and verified only after question selection.
  • MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis (magicbeanstalk): MBS extracts reasoning blueprints from verified hard seeds, decomposes them top-down, measures whether solved children improve parent accuracy, synthesizes harder novel leaves, and composes them bottom-up according to the original dependencies. Independent validation must preserve the blueprint and meaningful novelty, then the target policy must have mixed successes and failures with at least one complete, correct solution within budget. Not reproduced for now due to cost; validation pending, no executable entry.
  • CoBA-RL (coba-rl): Standard RLVR may reweight trajectories already accessible to the base model, raising pass@1 without extending the capability boundary visible at high pass@k. CoBA-RL estimates that boundary with repeated sampling, introduces teacher reasoning near or beyond it, and reinforces it with RL.
  • SPIRAL (spiral): The same model plays problem setter and solver. Automatically scored zero-sum multiround games create a progressively harder curriculum.
  • INTUITOR (intuitor): The direction is uniform-to-policy, not negative policy entropy. Confidence does not imply correctness.
  • Absolute Zero (absolute-zero): A proposer generates verifiable tasks, a solver attempts them, and an executable verifier supplies rewards. Adapt task difficulty to the current capability boundary.

Process and token credit assignment

  • Progress Reward Model for Reinforcement Learning via Large Language Models (progress-reward): It uses the subtask's semantics as a state prior and the difference in progress between successive states as dense reward for low-level RL. The plan is generated in advance; the LLM does not run at every environment step. Bounded component only; full experiments are not reproduced.
  • Inject a Turn, Not a Token: Learning Where to Redirect LLM Reasoning for Self-Correction (upturn): The following redraw explains the mechanism; it is not the original figure.
  • SPEAR: Distilling Domain-Adaptive Reasoning Skeletons via Sequential Symbolic Alignment in Reinforcement Learning (spear): Outcome rewards are sparse and neural process reward models are expensive. Project teacher reasoning into domain-specific symbolic milestones and reward student traces with order-sensitive longest-common-subsequence F1.
  • SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning (srpo): SRPO asks the current model to write a short reflection patch from the complete trajectory and environment outcome, then adds that patch to the original prompt. The same model acts as a teacher with this privileged context and gives per-token scores to the student's on-policy rollout.
  • ADRS (adrs): ADRS normalizes teacher scores within each interaction step, gates them by correlation with realized returns, then injects the gated token signal into the native reward-to-advantage path. No skills are needed at inference.
  • CoRT (cort): Replay the same response with a rubric and with criteria removed. Token likelihood differences redistribute GRPO's response-level advantage.
  • TCR (tcr): TCR constructs a checklist per example and subtracts an exponential moving outcome baseline from process scores. This concentrates updates on reasoning gains unexplained by outcome reward.
  • TACO (taco): A correct trajectory's uniform positive advantage can reward implausible low-probability tokens. Estimate contextual tail risk, smoothly reduce high-risk positive credit, and retain negative credit.
  • Math-Shepherd (math-shepherd): Sample several continuations from an intermediate reasoning step. Their final answer correctness supplies automatic step labels for verifier training and reranking.
  • Let's Verify Step by Step (process-supervision): A step-level reward model judges each reasoning step and prioritizes uncertain steps for annotation. The local comparison uses the same candidates and budget as outcome-only reward.

Reward construction and ranking signals

Multimodal and structured rewards

Distillation and training feedback loops

On-policy and context distillation

Advisors and targeted self-distillation

Teacher anchors and SFT–RL hybrids

  • CSD: Common-Support Distillation for Cross-Tokenizer Knowledge Transfer (csd): For 1:1 use shared token events; for 1:N or M:1 hold observed prefixes fixed and enumerate equivalent final completions; M:N defaults to the observed sequence only. The REST event preserves unmatched mass. Bounded component only; full experiments are not reproduced.
  • EliteReplay: Preserving Diverse Solutions in RL Post-Training via Quality-Diversity Search (elite-replay): Cells are indexed by question, approach, and structural descriptor, retaining only the shortest verified solution per cell. It is not an ordinary IID replay buffer. Bounded component only; full experiments are not reproduced.
  • Training-Free On-Policy Distillation (tf-opd): The key is not arbitrary covariance: use second moments of the inputs each teacher actually sees on its own generated responses after termination and loop filtering. Normalize by token count so longer answers do not dominate other domains. Bounded component only; full experiments are not reproduced.
  • Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation (road-mopd): Road uses the non-routed teacher to obtain compatibility evidence on the same student trajectory in the target domain. It adjusts token credit, computes an actual auxiliary gradient, and projects the target gradient while retaining a lower bound on its norm.
  • CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation (cast-opd): CAST aligns actual decoded text, projects the teacher span's total log probability uniformly onto the student span, then allocates credit by uncertainty and local smoothness. On unaligned positions, a KL anchor to the initial student limits drift.
  • Continually Post-training Small Models via Synthetic Self-distillation (synthself): The evaluated student cannot see reference answers. Text filtering alone is not training.
  • Learning More by Training Less: Uncertainty-Guided Progressive Contraction for Efficient Reasoning Post-Training (progressive-contraction): The paper samples Monte Carlo continuations from the actual prefixes, contracts the teacher pool by branch uncertainty, runs a real student probing epoch, and then selects and reroutes top-M examples by current student NLL. Teacher profiling cost and student exposure must be accounted for separately.
  • CompassOPD: Cross-Family On-Policy Distillation via Within-Family Likelihood Shifts (compass-opd): Absolute probability scales differ across model families. Compare the post-trained teacher against its reference, then center the change within its family to remove a global offset while retaining the direction.
  • Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation (oprd): Extract a weak model's before/after post-training logit direction and amplify the strong model's verifiable-reward gradient along it. Learning direction may transfer better than final capability.
  • Preserving General Capabilities during Domain Specialization with Uncertainty-Calibrated MOPD (uc-mopd): Ordinary MOPD seldom samples tokens with strongly positive advantages and cannot tell whether update directions are reliable. This method broadens the sampling temperature range, selects trajectories by positive-advantage density, and uses entropy-calibrated CLL probabilities to gate token updates.
  • From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection (grin): When on-policy sampling rarely finds new correct answers, inject golden responses into failing groups and train with mixed-policy importance correction. Return to on-policy exploration when performance improves.
  • Distilled RL (distilled-rl): RL provides sequence-level rewards, while OPD imitates a teacher unconditionally. Distilled RL uses the teacher-to-student probability ratio to reweight reward at each token, activates teacher weighting only for positive advantages, and uses a sequence geometric mean to remove length-scale bias.
  • ARMOR (armor): ARMOR actively samples anchor trajectories from a frozen reference and mixes them with current-policy rollouts. Data, rather than an auxiliary KL term, stabilize long-horizon RL.
  • MOPD (mopd): Joint RL couples domains; merging parameters or off-policy tuning can lose capabilities. Train specialists independently, then mix dense teacher signals only on student rollouts so domains can evolve in parallel.
  • CHORD (chord): Separate SFT and RL stages can overfit expert data or forget it too early. Add expert SFT as an annealed auxiliary term during on-policy RL; token-level uncertainty weights smooth the transition from imitation to exploration.

Other

LLM evaluation and calibration

Multimodal contextual preference calibration

Calibration-aware RL

RLVR

OPD

Online reinforcement learning

Preference optimization

Multilingual post-training

RLHF / RLVR

post-training

External rollouts

Asynchronous RL

RL training acceleration

Post-training data selection

Low-rank post-training

Multi-agent systems

Post-training algorithm components

Decoupling preference and optimization scales in DPO

  • Disentangling Optimization Scale from Preference Scale in DPO (normalized-dpo): Standard DPO beta changes preference noise and gradient size simultaneously, implicitly scaling learning rate. Divide a centered softplus by beta to preserve the minimizer and keep nonvanishing gradient in the beta-to-zero limit.

Negative self-distillation

rlvr

Online distillation

Multimodal post-training

Diffusion language model post-training

  • On-Policy Residual Learning for Diffusion Language Models (orl-diffusion): Training must see residual trajectories induced by the model itself. ORL combines online collection, state transfer, exact replay, and entropy contraction to narrow the residual-state distribution gap between training and inference. Not reproduced due to cost (review complete); research material only, no executable entry.

Reinforcement learning

On-policy distillation

RLVR and preference optimization

  • POLARIS: Guiding Small Models to Write Long Stories (polaris-hri): The same external rubric grades the policy and reference. Choosing the exact clipped branch in log-ratio space avoids overflow leading to apparently finite forward values and NaN gradients without changing the objective. Bounded component only; full experiments are not reproduced.

Reward modeling and calibration

Lookahead preference trees

GRPO

RL

Calibration rewards

Cross-domain RLVR capability fusion

  • Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms (rlvr-fusion): The paper compares three cross-domain capability fusion paradigms that reuse different artifacts: Merge combines expert task vectors, Mix RL combines training data, and MOPD reuses both experts and data. Average differences are at most 1.4 points but can reach 8.6 points on individual tasks, so the choice depends on expert availability, data, and cost.

Reward modeling and evaluation protocols

  • What do Reward Models Memorize? (rm-memorization-protocol): User-level grouping prevents pairs from one person crossing train/test. The local component aggregates real scores/manifests supplied by its caller and checks user consistency; it neither generates 25 reward models or the optimized data splits nor implements the underspecified co-occurrence search. Bounded component only; full experiments are not reproduced.

Self-review

  • Shockingly Simple Self-retrospection Improves Agentic Models Without RL (roft): Instead of directly updating the action policy with environment rewards, ROFT generates a self-retrospection after the complete trajectory and feedback, then supervises retrospection tokens only. The resulting training signal can inform future behavior without treating environment text or prior actions as targets.

Offline SFT

Rubric RL

Multi-reward RL

Reward modeling and credit assignment

  • SCAR: Shapley Credit Assignment for More Efficient RLHF (scar-shapley-credit): The local scope is exact enumeration for N<=10 and reward scatter at gamma=1. It does not treat an unspecified parser, an Owen hierarchy, or a synthetic reward model as the original system. Bounded component only; full experiments are not reproduced.

RL post-training

  • Sharpening Tax in Post-Training (sharpening-tax): Compare coverage of the base and post-trained policies under the same sampling budget. Posterior-tempered group sampling adjusts temperature by estimated difficulty to balance pass@1 and coverage.

Training workflows and resource allocation

Test-time adaptation

Multi-turn agent alignment

Video generation

Privileged self-distillation with evidence frames

Online RL and training stability

Trust regions, clipping, and gradient stability

  • Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence (gppo-framework): All combinations optimize tokens from real rollouts. This is not a registry alias for GRPO and does not replace the existing GPPO from arXiv 2508.07629.
  • Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization (erpo): ERPO observes that the induced query distribution also drifts during RL, so it puts regularization on the input side. Frozen-reference query likelihood gives static query weights; Query-KL controls environment-distribution drift without directly constraining the response score function.
  • GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs (gcpo): GCPO fixes the top-k singular spaces on both sides and restricts low-rank updates to their input and output orthogonal complements. This is a hard feasible set, not a soft penalty with a tuned coefficient.
  • RIPO (ripo): Fixed PPO ratio bounds can be too conservative at low probabilities and too loose at high probabilities. Use Fisher–Rao policy distance to set clip radius from the old action probability for a more even local KL budget.
  • STARE (stare): Identify entropy-critical tokens by batch surprisal quantiles, reweight their advantages, and use target entropy in a feedback gate.
  • GPPO (gppo): This can suppress exploration and learning from negative samples. GPPO keeps PPO's forward clipped objective while restoring backward signals in those regions with a stop-gradient boundary weight.
  • CISPO / MiniMax-M1 (cispo): Sample with a fixed rollout policy and compute token-level importance ratios. Clipping only the ratios preserves the direction of advantages and effective gradients.
  • VAPO (vapo): Critic bias, heterogeneous lengths and sparse rewards destabilize value-based PPO on long traces. VAPO pretrains a value model and adapts GAE and actor updates to response length.

Step rewards and credit assignment

Sequence objectives, length, and aggregation bias

  • Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL (tapo-trajectory): TAPO uses a geometric mean of prefix likelihood ratios, tightens clipping by length, and detaches the prefix so that earlier tokens receive no gradient from the current action. This adapter does not replace arXiv 2607.27973's TAPO.
  • RLCPR: Rethinking Probability-Based Reinforcement Learning From Posterior Concentration (rlcpr): RLCPR applies relative length penalty only if all responses are long and their posteriors concentrated. Mean direct-answer token entropy stratifies data so retained examples favor low uncertainty.
  • ReCo (reco-grpo): GRPO can resample high-probability responses and amplify saturated tokens, reducing reasoning diversity at large k. Weight responses inversely by expected occurrence in the rollout group, then use a Bernoulli variance ratio to focus updates on unsaturated decisions.
  • ConSPO (conspo): Express preferences among sequences in the same group as length-normalized InfoNCE, avoiding length and within-group scale biases caused by summing token scores.
  • LUSPO (luspo): Decompose RLVR objectives to explain different response length trajectories. GSPO still carries length bias; normalize sequence log-probabilities to prevent collapse.
  • GSPO (gspo): GSPO averages the log-ratios for a response, then exponentiates the result to get one sequence ratio. The entire sequence shares the clipping weight.
  • Dr. GRPO (dr-grpo): GRPO's response-length averaging and within-group standard deviation can introduce length and problem-difficulty biases. Remove both normalizations while keeping centered group-relative reward so trajectories retain their scale.

Agentic RL and teacher scheduling

PPO, REINFORCE, and group RL

  • Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards (ggpo): GGPO measures deviation by the difference between log-mean-exp and the arithmetic mean. It does not multiply updates by learner/behavior importance weights and periodically refreshes the reference to change subsequent targets. Bounded component only; full experiments are not reproduced.
  • When KL Regularization Misfires in Group Policy Optimization (zcpo): Independent KL penalties may update policy despite tied rewards or canceling reward gradients. Center response-average conditional KL within the group and put it in the frozen reward coefficient, not a standalone loss.
  • LUFFY (luffy): Put high-quality offline reasoning and online rollouts on the same support. Regularize importance ratios to retain on-policy behavior.
  • DAPO (dapo): Standard symmetric clipping can suppress exploration in online RL for long chains of thought, and groups with all correct or all wrong answers offer no learning signal. DAPO's Clip-Higher gives more room to increase probabilities; Dynamic Sampling discards zero-variance groups; token-level loss handles responses of different lengths fairly; and overlong responses receive a piecewise penalty.
  • RLOO (rloo): RLOO treats a full response as one action. Sample several responses for the same prompt and use the other responses' mean reward as the current one’s baseline.
  • DeepSeekMath / GRPO (grpo): GRPO samples a group of responses to each question and computes advantages from the group's reward mean and standard deviation, removing the critic. The policy still uses an old-policy ratio, clipping, and reference KL.
  • ReMax (remax): LLM rollouts have fast simulation, deterministic token transitions and typically terminal rewards. Drop PPO's value model and subtract the current policy's greedy reward as a prompt-dependent baseline for lower-variance REINFORCE.
  • InstructGPT / PPO-RLHF (ppo-rlhf): Increasing language-model size alone does not guarantee alignment with user intent. InstructGPT establishes the three-stage pipeline: SFT on annotator demonstrations, reward-model training on pairwise rankings, and PPO policy optimization with a KL penalty to limit drift from the SFT/reference model.

Advantage estimation and multi-objective optimization

  • Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation (probe-erpo): Estimate question reliability via a probe consistency ratio (PCR) without using final answers. Reinforce high-consistency samples, penalize lower-ranked candidates at low consistency, and constrain exploration with entropy.
  • Boosting LLM Exploration via Weak-Model Guidance in RLVR (weak-guide-rlvr): A smaller, weaker model supplies part of the reasoning prefix, exposing the target model to unfamiliar trajectories. Entropy truncation and mixed training on native and guided samples preserve coverage.
  • TTPO: Test-Time Policy Optimization (ttpo): Majority-vote pseudo-labels can be wrong, but rollouts disagreeing with the majority are usually still incorrect. TTPO applies OPSD to agreements and grouped RL to disagreements, separately filtering converged tokens and high-confidence errors.
  • GPRL (gprl): GPRL first computes a group-relative advantage separately for each preference dimension, then aggregates by context. A drift controller detects whether any one dimension dominates training and adjusts the weights.
  • TTRL (ttrl): Sample several answers for the same test question, use the majority answer as a pseudo-label, and update immediately without gold labels.
  • REINFORCE++ (reinforce-plus): Prompt-local standard deviations in GRPO/RLOO can randomly reweight groups of different difficulty. REINFORCE++ retains within-group centering but normalizes by a global advantage scale across batches to reduce variance and local bias without a critic.

Preference modeling and supervision

Pairwise, pointwise, and listwise preferences

  • CleanRM: Reward Modeling from Noisy Feedback with a Few Clean Feedback for RLHF (clean-rm): It performs optimal transport on the joint distribution of semantics and recovered preferences instead of matching only preference histograms. It then freezes the estimator and trains a reward model with a debiased surrogate. Bounded component only; full experiments are not reproduced.
  • Mind the Gap: Structure-Aware Consistency in Preference Learning (sa-dpo): SA-DPO sets the margin as a function of response distance while preserving implicit reward differences relative to a reference policy. It studies consistency among the margin, model capacity, and surrogate loss tail rather than merely tuning DPO's learning rate. Not reproduced due to cost (review complete); research material only, no executable entry.
  • SimPO (simpo): DPO keeps a reference model and sequence probability tends to favor short responses. SimPO uses mean token log-probability as the implicit reward, drops the reference, and adds a fixed margin to Bradley–Terry preference training.
  • ORPO (orpo): ORPO combines the chosen response's NLL and a chosen-versus-rejected odds-ratio penalty into one objective. Near probabilities of 0 or 1, odds provide a more sensitive contrastive signal than raw probability differences.
  • KTO (kto): It uses the expected KL between policy and reference as a reference point. Examples of each class can be collected independently and need not be balanced.
  • IPO (ipo): The paper proposes \(\Psi\)PO, which optimizes preference probabilities directly. The identity mapping yields IPO: fit a finite target margin instead of indefinitely widening the chosen/rejected gap with logistic loss on separable training data.
  • DPO (dpo): Conventional RLHF fits a reward model and then optimizes the policy with PPO, making training complex and unstable. DPO starts from the optimum of KL-regularized RLHF, writes the implicit reward as a policy-to-reference log ratio, and reduces training to binary classification on preference pairs.
  • SLiC-HF (slic-hf): Instead of a multi-model PPO-RLHF stack, enforce a margin between preferred and rejected sequence log-likelihoods while regularizing with SFT. The objective can use off-policy preferences collected for other models.
  • RRHF (rrhf): PPO-RLHF coordinates policy, old policy, reward and value models, making training and tuning complex. RRHF samples responses from several models or humans, uses reward to rank them all, aligns the model's average log-likelihood order with reward order, and applies SFT to the highest-quality answer.

Safety alignment and controllable supervision

  • SteerLM (steerlm): Traditional RLHF reduces multidimensional preferences to one reward, and users cannot change the target at inference. SteerLM uses an attribute predictor to label responses for helpfulness, quality, and other properties, adds attributes and values as conditions in ordinary SFT, and lets the user specify target attributes at inference.
  • RLAIF (rlaif): An AI labeler judges candidates in both presentation orders. Average away positional bias and train the preference policy with the paired labels.
  • Constitutional AI (constitutional-ai): In stage one, the model critiques and rewrites its responses according to the principles, then learns by SFT on the revisions. In stage two, AI compares responses, a preference model is trained, and its reward drives RLAIF.

Model merging and inference efficiency

Best-of fine-tuning and self-play

  • Self-Rewarding LM (self-rewarding): The current model generates candidates and judges them each round. New preference pairs feed the next DPO step, forming a bootstrapping loop.
  • SPIN (spin): Starting from an SFT model, SPIN uses the previous model to answer training prompts, treats human demonstrations as positives and self-generated answers as negatives, and trains the next model with a self-play discriminative objective. This repeats without new human preference labels.
  • RAFT (raft): RAFT samples multiple responses from the current model each round, ranks them with a reward model, discards lower-quality samples and maximum-likelihood fine-tunes on the selected responses. The updated policy starts the next round.

Training–inference consistency and efficient rollouts

Importance sampling and engine mismatch

  • KPop (kpop): A fixed ratio mask can reject legitimate exploration or keep harmful gradients. Collapse the sampled token and remaining vocabulary to a Bernoulli distribution; retain its update only if forward and reverse binary KL meet the threshold.
  • Missing Old Logits (missing-old-logits): Losing historical training-side logits conflates train/inference mismatch correction with staleness correction. Fixes include snapshots, an old-logit model, interrupted synchronization, and revised PPO-EWMA.
  • Online IcePop (online-icepop): Online IcePop allows only one update per batch, so the stale-policy ratio is 1 and PPO ratio/clipping are removed. Keep the IcePop two-sided mask and original in-range engine-mismatch ratio.
  • Stabilizing RL with LLMs (minirl): Separate train–inference mismatch from policy staleness. Use on-policy importance correction and combine off-policy clipping with MoE Routing Replay.
  • IcePop (icepop): One-sided TIS may retain very small mismatched ratios. Mask ratios outside a fixed two-sided interval and retain original corrections inside.
  • TIS (tis): TIS multiplies policy gradients by the training-to-rollout probability ratio. A one-sided upper cap controls heavy-tail variance while retaining small-weight samples.

Budget allocation and speculative rollouts