SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation¶
Candidate-policy objective, not paper-scale training.
Paper information¶
| Field | Details |
|---|---|
| Paper | Paper or original source |
| Institution | Alibaba / Qwen; first author Wen Wang. |
| First public release | 2026-08-04 (arXiv v1) |
| Author code | No public author code found as of 2026-08-24. |
| Local adapter or method key | smopd |
| Local reproduction code | src/auto_research/post_training/historical_b08_b09.py |
Paper method and evidence¶
Train specialist teachers with reward priorities, then merge their capabilities on the student's own trajectories.
flowchart LR
A[Train priority specialists] --> B[Generate student trajectories]
B[Generate student trajectories] --> C[Merge teacher guidance]Original paper figure¶
The original figure shows specialize-and-merge distillation. From the original paper; copyright remains with the authors.
Core equation¶
\[
\mathcal L=\sum_k\alpha_kD_{KL}(p_{T_k}\Vert p_\theta)
\]
Reported results¶
The paper beats GDPO for complementary and conflicting rewards on 1.5B/3B/7B backbones. No online inference.
Local reproduction¶
120 updates: accuracy 0.1953→0.3125 (+0.1172). Metrics, Batch index. Before/after arithmetic-smoke results on one candidate policy are not matched comparisons across methods. No full checkpoint, data, compute, multi-seed or benchmark. Raw outputs stay in runs/post-training/; experiment artifacts contain metrics only.