StepPO: Step-Aligned Agent Policy Optimization¶
Core mechanism evidence. Paper and local evaluation are separate.
Paper information¶
| Field | Value |
|---|---|
| Paper | StepPO (arXiv 2604.18401) |
| Affiliations | University of Science and Technology of China |
| First public release | 2026-04-20 |
| Authors' code | No official code found. |
| Adapter / method | steppo |
| Local code | src/auto_research/agent_research/ |
Original paper¶
Background and contribution¶
The environment observation/action step is the natural agent decision unit, unlike token-level MDP credit. Estimate values and GAE at step boundaries, then aggregate token ratios within a step before clipping.
flowchart LR
I[Environment step] --> V[Boundary value and GAE] --> R[Aggregate token ratios] --> C[Step-level clip]Key figure from the paper¶
Original Figure 3: StepPO. From the paper; copyright belongs to the authors. Not a local result.
Core mechanism¶
\[
\hat A_t^{\rm step}=\sum_{l\ge0}(\gamma\lambda)^l\delta_{t+l},\qquad r_t^{\rm step}=\exp\!\left(\frac1{|\mathcal T_t|}\sum_{j\in\mathcal T_t}\log r_j\right).
\]
Paper results¶
The paper reports consistent gains over token-centric RL on multi-hop QA, paper search, and text-world actions. No online A/B.
Local reproduction¶
PlanBench mini applies boundary values, step GAE, and step ratio clipping at every action and records counts. Metrics.
Reproduction limits¶
Actions are deterministic tool plans, not generated LLM tokens. Tests validate step alignment, not full-scale agent training.