ARMOR: reference anchor rollouts¶

简体中文

Isolates the RL update on a public candidate policy or deterministic agent mini-suite. The lightweight experiment does not establish paper-scale model or benchmark outcomes.

Paper information¶

Field	Details
Paper	ARMOR: reference anchor rollout (arXiv 2607.10481)
Organization	University of Science and Technology of China / Peking University
First public release	2026-07-11
Authors' code	No official code was found.
Adapter / method	`armor`
Local reproduction code	`src/auto_research/post_training/`

Original paper¶

Motivation and method¶

Reverse KL penalizes departure passively but does not ensure that useful reference solutions remain covered. ARMOR actively samples anchor trajectories from a frozen reference and mixes them with current-policy rollouts. Data, rather than an auxiliary KL term, stabilize long-horizon RL.

flowchart LR
    R["reference policy"] --> A["anchor rollouts"]
    P["current policy"] --> O["on-policy rollouts"]
    A --> M["mixed optimization"]
    O --> M
    M --> U["stable update"]

Key figure from the paper¶

Figure 4: Training flow and key optimization steps. From the original paper; copyright remains with the authors.

Core equation¶

\[ \mathcal L=\mathbb E_{\tau\sim\pi_\theta}[\ell_{\rm on}(\tau)]+\lambda\mathbb E_{\tau\sim\pi_{\rm ref}}[\ell_{\rm anchor}(\tau)]. \]

Paper results¶

On reasoning benchmarks, anchor rollouts mitigate validation collapse and support longer training. No production online A/B test is reported.

Local reproduction¶

Sample half of each training group from the current policy and half from the frozen reference. Record anchor counts and weights and final policy metrics separately.

auto-research post-train --algorithm armor --dataset gsm8k-candidate --maximum-examples 256 --steps 120 --seed 42

See the fixed-seed aggregate metrics.

Reproduction boundaries¶

The reference is the locally initialized candidate policy. No paper-scale long-horizon training or real trajectory pool is included.