SimPO: Simple Preference Optimization with a Reference-Free Reward¶

Real token-level preference loss on a small free-generation model, not paper-scale training.

Paper information¶

Field	Details
Paper	SimPO: Simple Preference Optimization with a Reference-Free Reward
Institution	Princeton University.
First public release	2024-05-23
Author code	princeton-nlp/SimPO.
Local adapter or method key	`simpo`
Local reproduction code	`src/auto_research/post_training/`

Paper method and evidence¶

DPO keeps a reference model and sequence probability tends to favor short responses. SimPO uses mean token log-probability as the implicit reward, drops the reference, and adds a fixed margin to Bradley–Terry preference training.

flowchart LR
  P[Preferred and rejected responses] --> R[Mean token log-probability reward]
  R --> M[Fixed-margin preference loss]

Original paper figure¶

The original figure shows the reference-free preference objective. From the original paper; copyright remains with the authors.

Core equation¶

\[ r_{\mathrm{SimPO}}(x,y)=\frac{\beta}{|y|}\log\pi_\theta(y|x),\qquad \mathcal L=-\log\sigma\!\left(r(x,y_w)-r(x,y_l)-\gamma\right). \]

Reported results¶

The paper reports up to +6.4 points on AlpacaEval 2 and +7.5 on Arena-Hard versus DPO. Gemma-2-9B-it reaches 72.4% length-controlled AlpacaEval 2 win rate. No production A/B.

Local reproduction¶

Train a freely generating causal LM without a reference model. Compare against other free-generation methods with the same warmup, data, steps and seeds.

auto-research post-train --algorithm simpo --dataset arithmetic-generate \
  --maximum-examples 48 --steps 6 --seeds 42,43,44 --offline

Stable metrics. No Mistral/Llama/Gemma-scale run or AlpacaEval judge.