SIPO: Unifying RLVR and on-policy self-distillation¶

L1 dual-context objective and policy gradient; no paper-scale rollouts.

Paper information¶

Field	Details
Paper	SIPO
Institution	University of Illinois Urbana-Champaign (first author).
First public release	2026-09-29 (arXiv v1)
Author code	Yueeeeeeee/SIPO.
Local adapter or method key	`sipo`
Local reproduction code	`src/auto_research/post_training/latest_20260930.py`

Paper method and evidence¶

Sparse outcome reward gives little token-level credit, while ordinary OPSD can overtrust its teacher or penalize long sequences. Construct positive and negative privileged teacher contexts for the same on-policy rollout. Their log-probability difference cancels shared bias. Environment reward sets the main direction; dense evidence redistributes credit across tokens.

flowchart LR
  O[On-policy rollout group] --> R[Environment reward]
  O --> P[Positive teacher context]
  O --> N[Negative teacher context]
  P --> D[Contrast log probabilities]
  N --> D
  D --> A[Token advantage]
  R --> A
  A --> G[Policy gradient]

Original paper figure¶

Figure 1 contrasts sparse RLVR and dense OPD supervision. From the original paper; copyright remains with the authors.

Core equation¶

\(e_t=\log q_t^+-\log q_t^-\) and \(A_t=A_{outcome}+\beta(e_t-\bar e)\). Even tied outcome rewards can have nonzero token signal without gold in student rollout.

Reported results¶

The authors report gains over RLVR and OPSD on reasoning and coding. Not local evidence.

Local reproduction¶

Three-seed results test all-failure groups with token signal, finite gradients, and retention of the outcome direction. Sanitized A100 receipt.