Beyond Token-Local Imitation: Reward-Compatible Temporal Credit Assignment for On-Policy Distillation¶
L1 mechanism diagnostic. Discounted credit and RBM execute; no paper-scale training.
Paper information¶
| Field | Value |
|---|---|
| Paper | Beyond Token-Local Imitation: Reward-Compatible Temporal Credit Assignment for On-Policy Distillation |
| Affiliations | Tsinghua University / DiDi Voyager Labs |
| First public release | 2026-09-15 (arXiv v1) |
| Authors' code | No public author repository found as of 2026-09-16. |
| Adapter / method | gamma-opd |
| Local code | src/auto_research/post_training/latest_20260916.py |
Original paper¶
Background and contribution¶
Approximate sequence-level reverse KL with discounted temporal credit and add verifiable outcomes through bounded reward advantages.
flowchart LR
I[Public input and state] --> C[Discounted credit] --> B[Bounded reward modulation] --> O[Weights or policy update]Key figure from the paper¶
Original Figure 1: temporal credit assignment. From the paper; copyright belongs to the authors. Not a local result.
Core mechanism¶
The reference preserves defining gates, weights, and state changes with auditable intermediates for matched-budget comparison. Three fixed-budget seeds test invariants, not paper-scale training.
Paper results¶
The paper's capability, speed, and cost findings are not local evidence or production gains.
Local reproduction¶
Three-seed metrics are diagnostic_only=true, not dashboard capability gains.
Reproduction limits¶
Discounted credit propagation and RBM only; no paper-scale LM training.