简体中文 · English
PR-OPD: Privileged representation self-distillation for agentic RL¶
L1 representation-alignment objective. Implements stop-gradient privileged alignment across layers and response tokens.
Paper information¶
| Field | Details |
|---|---|
| Paper | arXiv v1 |
| Institution | University of Florida (first author affiliation) |
| First public release | 2026-09-29 (arXiv v1) |
| Original source code | Yes: balibata/PR-OPD |
| Adapter | pr-opd |
| Local reproduction code | src/auto_research/post_training/latest_20260930_followup.py |
Background and main changes¶
After a GRPO warm start, the policy writes a hindsight skill for each completed trajectory. Run two forward passes on the same response with ordinary context and skill context. The latter is a stop-gradient teacher. PR-OPD aligns projected hidden states at each layer and response token, then combines alignment with the reward objective. Inference removes the skill and teacher branch.
Key figure from the paper¶
Figure 1 motivates representation alignment by contrasting sparse probability signals with widespread hidden-state changes. From the paper; copyright belongs to the authors.
Local reproduction¶
Tests cover zero loss for identical representations, positive loss for different ones, masking and teacher stop-gradient. See three-seed mechanism results. ALFWorld, WebShop and the GRPO warm start were not run. The paper's maximum +14.0 points cannot be claimed here.