Interaction learning: How are credit and teacher information assigned after standardizing trajectories?

简体中文 · English

Updated 2026-10-27 · 53 topic entries / 6 representative comparisons

Agent Lightning turns execution into trainable transitions. GiGPO, AdaStep, and DepGPO assign credit using repeated states, return variance, and command dependencies, respectively. Frontier-OPD and ASTEROID change how teacher data reaches students. Recording a trajectory is only the entry point; objectives and information access define the algorithms.

Mechanisms · Trade-offs · Cross-paper questions · Updates

How to read this card

The table compares representative methods. It is not a shared-benchmark leaderboard. Teaching examples are not measured results.

Mechanism comparison

Axes: Whether supervision units are tokens, environment steps, command dependencies, or expert trajectories · Whether local credit depends on recurrent states, stochastic returns, or trusted tracing · Whether teachers supply distributions or short bridges versus complete successful episodes · Whether privileged information stays on the training side or enters deployment input

Paper and source	Mechanism	Benefits and trade-offs
Agent Lightning · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Records common MDP transitions to decouple execution and training, then connects credit assignment.	A common interface does not specify every credit algorithm. Locally only a controlled patch policy updates, not an LLM trained with LightningRL.
GiGPO · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Full trajectory groups produce macro advantages; cross-trajectory step groups sharing an anchor state provide micro relative advantages.	Requires trustworthy state alignment and sufficient repetition. A shared plan step is only a local proxy for state convergence.
AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Uses the fraction of return variance explained by actions to adjust local advantage, reducing correction when within-action continuation is noisy.	No separately trained critic, but requires several actions and repeated samples per anchor. Synthetic local returns check estimator direction without policy updates.
Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agents · Original source Sections in the Chinese source: 背景与主要改动, 公式与关键区别, 复现边界与下一步	Traces verifier resources back through command reads and writes, assigning credit to direct or indirect writers and supporting reads while preserving token-weighted mean advantage.	Needs real traces and removal of verifier-produced resources, not guessing from shell text. Handwritten local command diagnostics are not Terminal-Bench training.
Frontier-OPD: Discovering the Edge of Agent Capability and Teaching Agents to Go Beyond · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 本地复现	Aggregates signed teacher–student disagreement with balanced trajectory weights to select intervention states. Short teacher bridges plus student continuations pass a length-matched agreement gate before only bridge responses enter training.	Ranking is not admission; agreement is not success, and validation continuation stays out of training. The A100 run checks only an L1 numeric protocol; full training is expensive.
Asymmetric On-Policy Distillation Learns In-Context Exploration · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 保留的定义机制与缩小边界	Collects student exploration history, then a full expert success on the same hidden task, teaching prediction of expert actions from student history.	BAMDP cross-episode conditioning differs from general POMDP takeover. The local three-door, one-step small-model training is not multistep robotics capability.

Trade-off analysis

Local credit is trustworthy only when its evidence is trustworthy

GiGPO complements global advantage with return comparisons within a shared anchor. AdaStep asks how much variation is explained by actions versus random continuation. DepGPO instead uses actual file, stdout, and verifier dependencies, without requiring recurrent states. These are not interchangeable step rewards. Fine-grained numbers do not add evidence without trustworthy states or traces. Agent Lightning's common transitions help connect algorithms but do not supply missing conditions.

Sources: GiGPO · AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning · Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agents · Agent Lightning

Teachers change training data, not what deployment can see

Frontier-OPD selects short intervention from student-visited states. Compare a window matching actual bridge length plus continuation, and add only teacher bridge responses. ASTEROID conditions on exploration history and then supplies a full success for the same hidden task, covering contexts outside smooth expert openings. Teacher units and objectives differ. Deployment sees only public observations and its own history, not hidden tasks, gold, or future outcomes inserted for convenient replay.

Sources: Frontier-OPD: Discovering the Edge of Agent Capability and Teaching Agents to Go Beyond · Asymmetric On-Policy Distillation Learns In-Context Exploration

Cross-paper questions

Try answering before expanding the explanation. Each question relates at least two papers.

When outcomes vary widely after the same action, why do GiGPO, AdaStep, and DepGPO need different evidence?

Show answer and example

Answer: GiGPO compares local returns across actions at one anchor. AdaStep uses within-action and total variance to assess reliability and reduces correction under high continuation randomness. DepGPO checks whether command outputs reach the verifier by dependencies rather than using within-action variance. Choose evidence according to reliable recurrent states or tracing; synthetic anchors cannot establish a fair real-state comparison.

Illustrative example: Incorrect answers after identical searches may reflect later writing variation. A terminal task can instead check whether a write command supplied data consumed by the final test.

Sources: GiGPO · AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning · Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agents

Why can't Frontier-OPD teacher bridges and ASTEROID expert successes both be reduced to one expert label at the current state?

Show answer and example

Answer: Frontier-OPD uses a short bridge to change reachable continuation and admits responses by equal-length agreement over bridge plus student continuation. ASTEROID conditions on student exploration before learning from a complete same-task expert episode; BAMDP preserves hidden tasks across episodes. A single label erases differences in context, collection units, and admission.

Illustrative example: Briefly helping a student past a blockage and returning control is not the same training example as a whole expert success after two student trials.

Sources: Frontier-OPD: Discovering the Edge of Agent Capability and Teaching Agents to Go Beyond · Asymmetric On-Policy Distillation Learns In-Context Exploration

When does connecting Agent Lightning transitions to GiGPO or Frontier-OPD count as training integration?

Show answer and example

Answer: Records must contain required state, sampling policy, and permitted supervision. The trainer must update parameters with the appropriate advantage or distillation objective and evaluate. GiGPO needs valid anchor groups; Frontier-OPD needs original histories, valid bridges, and training-pool isolation. Names, counts, and callbacks are insufficient. Local controlled patch updates and L1 contracts are not LLM policy training.

Illustrative example: A thousand actions without shared-state identity or frozen teacher/student probabilities cannot compute micro advantages or validate teacher bridge admission.

Sources: Agent Lightning · GiGPO · Frontier-OPD: Discovering the Edge of Agent Capability and Teaching Agents to Go Beyond

Update log

Updates record new approaches, changed trade-offs, and follow-up evidence.

2026-10-27 · Initial version: compared transition integration, state-group credit, variance reliability, command dependencies, and two teacher-data mechanisms, distinguishing training and deployment information.

Covers 53 topic entries; see the full list below.

All 53 topic entries

The full catalog follows. The main comparison covers the representative papers listed above.

• Frontier-OPD: Discovering the Edge of Agent Capability and Teaching Agents to Go Beyond
• Asymmetric On-Policy Distillation Learns In-Context Exploration
• RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution
• Evo2: Evolving the Evolver through Weight-Harness Co-Evolution
• When Should Agents Think? Adaptive Reasoning via Cross-Turn Estimation
• Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agents
• AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning
• GraphHCA: Closed-Form Hindsight Credit Assignment for Long-Horizon LLM Agents
• Dr.Credit: Rubric-Grounded Process Credit Assignment for Deep Research Agents
• Dependency-Aware Trajectory Refinement for Efficient Multi-Turn Agent Fine-Tuning
• HarnessBandit: Joint Learnability-Transferability Scheduling for Multi-Harness Agentic Reinforcement Learning
• T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks
• What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents
• What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents
• SPO++: Stabilizing Asynchronous Agentic Reinforcement Learning via Measure-Theoretic Token Correction
• AHEAD: Agentic Hints for Effective Agent Development
• AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces
• Agent-G²: Gaussian Guidance for Agentic Reinforcement Learning
• AUSO: Action-Level Unified Skill Optimization from Internalization to Utilization
• SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning
• SPADE: Self-Play in Adaptive Synthetic Executable Environments
• RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training
• PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs
• TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agents
• Efficient Reinforcement Learning for Long-Horizon Tool-Use Agentic Tasks
• ResidencyRL: Reinforcement Learning in Simulated Clinical Environments
• EnvACE
• Contextual Information Policy Optimization for Search Agents
• AgentOPSD
• State2State: Environment-Derived Mid-Training for LLM Agents
• OCSD
• EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents
• HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learning
• TAPO
• Group-Reflective Self-Distillation
• CAST
• SEED
• TurnOPD
• StepPO
• SEARL
• Reasoning-Driven Synthetic Data Generation and Evaluation
• Dual-Modality Multi-Stage Adversarial Safety Training: Robustifying Multimodal Web Agents Against Cross-Modal Attacks
• PEARL
• Agent0
• Agent-R1
• MUA-RL
• Agent Lightning
• WebAgent-R1
• GiGPO
• RAGEN
• DeepResearcher
• Search-R1
• LOOP

Paper-by-paper questions · All deep dives