Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress¶
Core candidate-policy objective.
Paper information¶
| Field | Details |
|---|---|
| Paper | Paper or original source |
| Institution | No affiliation disclosed; first author Chen Yang. |
| First public release | 2026-08-19 (arXiv v1) |
| Author code | No public author code found as of 2026-08-24. |
| Local adapter or method key | r2-opd |
| Local reproduction code | src/auto_research/post_training/historical_b08_b09.py |
Paper method and evidence¶
Rank reasoning spans by teacher reward and an independent progress reward. Mask distillation where those rankings conflict.
flowchart LR
A[Score spans with teacher] --> B[Compare independent progress]
B[Compare independent progress] --> C[Mask conflicting supervision]Original paper figure¶
The original figure shows reasoning-progress filtering. From the original paper; copyright remains with the authors.
Core equation¶
\[
A_i^{R^2}=A_i^{OPD}\mathbf1[\operatorname{rank}(r_i^T)=\operatorname{rank}(r_i^P)]
\]
Reported results¶
The paper reports consistent gains over standard OPD, especially on harder reasoning tasks.
Local reproduction¶
120 updates: 0.1953→0.6641 (+0.4688). Arithmetic metrics and batch index. Before/after on one candidate policy is not a fair comparison with other methods. No author checkpoints, full data/compute, multiple seeds, or benchmark; runs/post-training/ stays local and experiment artifacts retain metrics only.