Interactive policy learning: converting trajectory feedback into updates¶
简体中文 · English
Agents / 212 questions · 53 papers
Try answering each question before opening the explanation. The bank grows as papers and questions are added.
Read this topic's comparisons, analysis, and cross-paper Q&A
212 questions · 53 papers
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does Frontier-OPD ask teachers to take over only at selected states instead of generating all training trajectories?
Frontier-OPD: Discovering the Edge of Agent Capability and Teaching Agents to Go Beyond
Show answer and example
Answer: Full teacher interaction is expensive, and imitating smooth teacher trajectories may miss student bottlenecks. Frontier-OPD finds teachable frontiers from student-visited states, briefly bridges with the teacher, then hands control back. Savings must be accompanied by verifying that the student can continue after the bridge.
Illustrative example: If the student repeatedly gets stuck opening a door after finding its key, test a short teacher intervention there rather than having the teacher finish every task from scratch.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why average student–teacher disagreement within trajectories and then across trajectories instead of across node visits?
Frontier-OPD: Discovering the Edge of Agent Capability and Teaching Agents to Go Beyond
Show answer and example
Answer: Loops repeatedly visit states; counting each visit as independent evidence lets one circling path dominate. Two-stage averaging balances trajectory weight. Representative histories then support real teacher-bridge trials. Signed disagreement for ranking differs from agreement for validation; a high node score does not guarantee takeover helps.
Illustrative example: Three visits at disagreement ten in one trajectory and one zero in another give a balanced score of five. Visit averaging gives 7.5 and overweights the loop.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
Why use different criteria for high-ranking frontier nodes and valid teacher bridges?
Frontier-OPD: Discovering the Edge of Agent Capability and Teaching Agents to Go Beyond
Show answer and example
Answer: Signed teacher–student disagreement prioritizes where to spend limited teacher budget. Bridge validation compares actual takeover plus student continuation to an equally long historical window, requiring valid replay and nondecreasing agreement. Prioritization is not training-data admission; a high-ranked node does not prove teaching works.
Illustrative example: High disagreement may warrant a trial even if the student still cannot continue after takeover. Do not add that failed bridge to training merely because it ranked high.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
How are windows compared when the bridge is shorter than K, and can its continuation enter training?
Frontier-OPD: Discovering the Edge of Agent Capability and Teaching Agents to Go Beyond
Show answer and example
Answer: Compare equal windows of actual bridge length plus H. Require valid replay, nonterminal handoff, and complete continuation; accept bridge responses only when agreement does not decrease. Student continuation used for validation cannot enter training, and retain repeated responses separately. The local path is only the L1 numeric protocol, without the trainer or full experience pool.
Illustrative example: If K caps the bridge at two turns but the teacher uses one, compare windows of one plus H. Do not use a longer historical window or train on the validation continuation as teacher demonstration.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why might imitating expert successes be insufficient in a hidden-task environment?
Asymmetric On-Policy Distillation Learns In-Context Exploration
Show answer and example
Answer: The student tries imperfect actions and encounters information and states absent from smooth expert paths. ASTEROID combines student exploration with expert success to cover actual uncertainty. Exploration uses visible interaction, and expert training information remains isolated rather than exposing hidden tasks or solutions during evaluation.
Illustrative example: The expert takes the right path immediately, while the student tries the wrong door first. Learning only the expert's opening does not show the student how to use observed failure information.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why let the student explore before learning the expert's full successful trajectory for the same hidden task?
Asymmetric On-Policy Distillation Learns In-Context Exploration
Show answer and example
Answer: An expert knows the hidden goal, so a direct success omits the context of continuing after mistakes. Condition expert-action prediction on the student's actual history, teaching task inference from observations and rewards. The hidden goal stays unavailable during deployment, and student exploration actions are not automatically correct labels.
Illustrative example: After the student searches two wrong rooms, the expert's success on the same task can teach how to narrow the search from that history rather than merely memorize a direct opening.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
Why shouldn't cross-episode BAMDP conditioning and generic POMDP takeover be reduced to single-step DAgger?
Asymmetric On-Policy Distillation Learns In-Context Exploration
Show answer and example
Answer: A fixed-latent BAMDP accumulates task knowledge across episodes and pairs it with an expert episode on the same task. A generic POMDP hands over at an intermediate student state. Complete expert traces and context are defining; a single current-action label discards them. Deployment cannot keep invoking a latent-aware expert.
Illustrative example: If the hidden door stays the same across three attempts, first-round failure informs the next. If the door changes each round, the same cross-episode history no longer conveys that knowledge.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
How does the local single-step three-door diagnostic preserve complete expert trajectories, and what are its limits?
Asymmetric On-Policy Distillation Learns In-Context Exploration
Show answer and example
Answer: Each door choice is a complete one-step episode, so expert length one reflects the environment rather than general simplification of a multistep method. A real small Transformer is supervised only on the terminal expert query, not student-history actions. Deployment starts with empty history and no expert; there is no general POMDP or robotics comparison.
Illustrative example: An incorrect student attempt stays in context, but loss does not train it to repeat the error. The correct expert action is stored separately as the training target.
Sections in the Chinese source: 核心公式, 本地复现, 保留的定义机制与缩小边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does RSIPO update old skills as the policy improves?
Show answer and example
Answer: Older skills reflect older policies and can become redundant, obsolete, or worse than direct solving by the current student. RSIPO forms skills from current trajectories and tests same-start gains, allowing skills and policy to co-evolve. Assistance is not inherently beneficial; keep independent controls and do not include original answers in evaluation input.
Illustrative example: An early skill uses a long search while the current policy finds evidence directly. Re-measure instead of forcing an older, potentially slower skill.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why shrink estimated skill gains by reliability before setting skill-enhanced teacher distillation weights?
Show answer and example
Answer: Some skills are useless or worse than the student, so equal trust can transfer regressions. Independent controls and same-start replay estimate utility differences. Shrink noisy estimates toward the group mean and use more reliable gains to adjust distillation trust. Skill reward and shrinkage-based trust have distinct roles; invalid replay is not paired evidence.
Illustrative example: One lucky successful trial merits shrinkage because the gain is uncertain. A skill that works repeatedly in same-start controls can receive more distillation weight.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does dynamic skill–policy co-updating differ from a fixed offline skill library in freshness?
Show answer and example
Answer: A fixed library stabilizes retrieval conditions but preserves experience from old policies and tasks. Co-updating builds skills from current trajectories and tests benefits with independent controls and same-start replay, tracking changing ability. It adds nonstationarity and evaluation cost. Update counts are not benefit or evidence of a reliable teacher.
Illustrative example: After a policy learns a shorter evidence path, a new skill can reflect it. A fixed old library may continue recommending the obsolete path.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Does the local project have an RSIPO adapter, real skill replay, and policy training?
Show answer and example
Answer: No. It records the specification and cost decision only, with no paper adapter, metrics, or GPU pass. An implementation must preserve current-policy skills, valid same-start replay, independent controls, and training/test separation. Another skill operator or fixed rule cannot substitute for full RSIPO or Evolve integration.
Illustrative example: A working GRPO trainer does not establish skill utility estimation or reliability-weighted OPSD. Without those, the algorithm is not implemented.
Sections in the Chinese source: 核心公式, 本地复现, 边界与后续触发条件. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does Evo2 want proposal and reflection abilities to learn from search experience?
Evo2: Evolving the Evolver through Weight-Harness Co-Evolution
Show answer and example
Answer: A fixed evolver repeats the same proposal habits and may persist with low-value or invalid changes. Evo2 trains generation and reflection from search experience while freezing the task executor to clarify what updates. New search versions do not mean the base executor trained, and benefits still require independent execution evaluation.
Illustrative example: Train a proposer to learn from repeated invalid structures without necessarily changing the weights of the model that executes the tasks.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why does training the Proposer and Reflector with a frozen executor improve the evolver itself?
Evo2: Evolving the Evolver through Weight-Harness Co-Evolution
Show answer and example
Answer: The task model stays fixed, but parameters of the policies that propose and repair harnesses change. Effective edit trajectories provide training signals for both roles. Improvement targets how harnesses are changed, not just the candidates. Keep actions still earn zero; unchanged proposals are not successful innovations.
Illustrative example: The same executor handles tasks, but the trained proposer chooses better log compression and the reflector fixes bad hooks. The gain comes from changing the improvement process.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does tree-parent weight ε+|V−1/2| differ from favoring only high-success nodes?
Evo2: Evolving the Evolver through Weight-Harness Co-Evolution
Show answer and example
Answer: It favors values far from the midpoint and retains a fixed root sampling share. This allocates search rather than monotonically choosing higher scores. Cross-branch crossover and the reflector also produce new harnesses; greedy champion-only search is not equivalent.
Illustrative example: V=0 and V=1 are both farther from the center than V=1/2, so parents are not selected simply in descending success-rate order.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
How are keep rewards, role advantages, and truncated tokens handled in the loss?
Evo2: Evolving the Evolver through Weight-Harness Co-Evolution
Show answer and example
Answer: Keep earns zero. For each role, subtract the group mean reward. Normalize token loss by fixed Lmax and mask truncated outputs so length does not alter training scale. The real local path has specific environment and resource limits: a 7B policy/executor plus frozen copies is not presumed to fit A30 or constitute the full paper benchmark.
Illustrative example: Do not supervise an incomplete truncated tail as a correct edit. Averaging by the shorter actual length would also alter the original training weight.
Sections in the Chinese source: 核心公式, 本地复现, 结果、验证与边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does RACE remove reasoning that does not help an agent's trajectory?
When Should Agents Think? Adaptive Reasoning via Cross-Turn Estimation
Show answer and example
Answer: Long-horizon agents generate much reasoning that repeats information or does not affect later actions, yet adds training and inference load. RACE reduces redundancy while preserving behavioral support. Shorter is not inherently better: deletion can remove needed context, so check downstream effects rather than using word count alone.
Illustrative example: Repeating a goal without changing later choices may be redundant. A brief note with a critical path cannot be removed just because it is short.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why check likelihood changes for all later original actions when deleting a turn's reasoning?
When Should Agents Think? Adaptive Reasoning via Cross-Turn Estimation
Show answer and example
Answer: An unchanged immediate action does not show earlier reasoning is irrelevant to later planning. After tentative deletion, evaluate later actions in the affected context and proceed only if changes stay within tolerance. Accepted deletions persist for later compression. Checking only the current turn misses long-range dependencies, and removed token counts are not task success.
Illustrative example: Reasoning may record a safety restriction while the current action only reads a file. A later deletion relies on that restriction, so a current-turn-only check can wrongly remove critical reasoning.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does a reasoning loss mask differ from actually deleting training context?
When Should Agents Think? Adaptive Reasoning via Cross-Turn Estimation
Show answer and example
Answer: A mask removes supervision for deemed-removable tokens, while other actions train under prescribed conditions. Deleting context directly changes later prediction distributions. RACE also supervises early closing and keeps denominators for all original policy tokens and turns; rescaling by amount removed would silently change the objective.
Illustrative example: If half the reasoning supervision is removed, still normalize by original token count to avoid inadvertently doubling the weight of remaining actions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
How do real trajectory requirements and the local bounded-calculator run limit conclusions?
When Should Agents Think? Adaptive Reasoning via Cross-Turn Estimation
Show answer and example
Answer: Real training data need an environment, revision, split, and reasoning/action/observation records. QA rows are not tool trajectories. The local calculator actually samples actions and runs SFT/on-policy updates, but is L1 mechanism evidence only. A few successes under one seed do not replicate ScienceWorld or four-environment RL.
Illustrative example: Calling two real calculation tools demonstrates interactive training, not long-horizon web or embodied capability.
Sections in the Chinese source: 核心公式, 本地复现, 验证与边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why track command dependencies in DEPGPO when terminal tasks only yield final success or failure?
Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agents
Show answer and example
Answer: A long trace contains many file reads and writes; terminal reward cannot identify which commands produced the result. DEPGPO connects verification to earlier effective work through actual dependencies. Dataflow is not causal contribution of every command, and missing records or verifier side effects require explicit boundaries.
Illustrative example: Trace the source file read by a final test to the command that wrote it. Unrelated directory browsing should not get equal credit just because it appeared in the trace.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why connect file reads to the writers of the still-current versions in a command dependency graph?
Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agents
Show answer and example
Answer: Reads consume the version active at that moment. Earlier overwritten writes may not affect the final artifact. Trace backward from verifier-checked outputs to connect data-contributing commands to the result and avoid sharing terminal advantage with unrelated steps. Graph construction needs trusted execution records, not guesses from command text.
Illustrative example: If a configuration is written and overwritten before a test reads it, the unused old write should not receive the same credit as the current version.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does dependency credit differ from uniform trajectory weighting in GRPO?
Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agents
Show answer and example
Answer: Unit weights do not distinguish supporting actions, final writes, and unrelated commands. DepGPO distributes step credit by traceable writes and distance, then normalizes to preserve token-weighted mean advantage and sign. It depends on tracer accuracy and does not prove causality or individual necessity.
Illustrative example: An input read followed by an answer write can get indirect credit. Unrelated temporary-file cleanup gets less weight, while negative trajectory advantage stays negative.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
What boundaries apply to multiple commands per step, missing records, and verifier-generated files?
Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agents
Show answer and example
Answer: Sum command credits within one step, rather than averaging. Missing records or fully tied weights revert to unit weights. Exclude verifier-written resources and stdout values already present in the task prompt; paths and line numbers come from trusted tracing. Local work parses no shell, runs no real Qwen RL, and only verifies the mechanism on four hand-built chains.
Illustrative example: If the grader generates a result file, treating it as an agent write rewards a nonexistent contribution. Remove such artifacts from the tracer input.
Sections in the Chinese source: 公式与关键区别, 本地复现（机制验证）, 复现边界与下一步. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does AdaStep distinguish an action's effect from luck in later trajectories?
AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning
Show answer and example
Answer: At the same intermediate state, later decisions still affect success. Assigning terminal differences directly to the first action is noisy. AdaStep uses multiple observed trajectories to assess local credit reliability, retaining global outcomes while cautiously adjusting step signals. It trains no separate critic and does not prove causal effects from few samples.
Illustrative example: After the same lookup, one continuation answers correctly and another transcribes incorrectly. Compare between-action differences with within-action fluctuations rather than blaming the lookup for all variation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why lower local credit correction when within-action return variance is high at the same anchor state?
AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning
Show answer and example
Answer: Random continuations after one action can produce very different outcomes, implying the local action explains little of the variation. Variance decomposition trusts local advantage more when between-action differences dominate; otherwise it preserves global advantage and suppresses the correction. Estimate weights from raw returns rather than arbitrarily normalizing away variance relationships.
Illustrative example: If rollouts from both actions A and B produce zeros and perfect scores, continuation randomness dominates. Do not strongly attribute a single outcome to either action.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does AdaStep's return-statistics approach differ from training a critic for step evaluation?
AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning
Show answer and example
Answer: A critic learns state or action values to organize credit. AdaStep estimates local-correction reliability directly from observed returns for different actions at the same anchor, with no separate critic. It saves a predictor but still needs repeated states and sample coverage. Critic-free does not imply noiseless or universally more accurate.
Illustrative example: Repeated actions from the same webpage state supply empirical variation. A single occurrence cannot provide a reliable local estimate by itself.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
How are few actions, few samples, zero variance, and normalized returns handled?
AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning
Show answer and example
Answer: Estimate the ratio only with at least two actions, two samples per action, and positive total variance; otherwise use the paper's w=1 fallback. Equal returns give zero local advantage. Weight statistics use raw returns, and groups include task IDs to avoid mixing names across tasks. No real environment or policy training ran locally.
Illustrative example: One action cannot reveal between-action variation, but that does not justify inventing w=0. Identical rewards should not create local correction out of nothing.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does GraphHCA give long-horizon actions a structured training signal before terminal success?
GraphHCA: Closed-Form Hindsight Credit Assignment for Long-Horizon LLM Agents
Show answer and example
Answer: Sparse terminal feedback can obscure early exploration and key transitions behind one trajectory reward. GraphHCA organizes observed transitions in a graph and propagates success to connected states to estimate action progress. The graph covers observed data only, its potential is not ground truth, and deployment actors cannot see training success labels in advance.
Illustrative example: If several routes reach success after opening a key door, the graph connects that experience. It cannot assume success behind an unseen door.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why solve a discounted state-graph fixed point before assigning action credit by differences in log potential?
GraphHCA: Closed-Form Hindsight Credit Assignment for Long-Horizon LLM Agents
Show answer and example
Answer: Propagate terminal successes and failures as boundary conditions over observed transitions to obtain discounted success potential. Differences in adjacent log potentials turn long-range evidence into local progress, then standardize across actions from a shared state. Complete rollouts support training attribution; do not expose hindsight success boundaries as observations to evaluated agents.
Illustrative example: Two paths through one intermediate state can share downstream success evidence. Compare whether actions leaving that state approach success or lead toward a dead end.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How do differences in graph potential differ from broadcasting one advantage to the whole trajectory?
GraphHCA: Closed-Form Hindsight Credit Assignment for Long-Horizon LLM Agents
Show answer and example
Answer: Trajectory advantage does not distinguish steps moving toward promising states. Potential differences highlight improved or worsened state value and are standardized among actions sharing a start. They depend on empirical transition probabilities and ε, not independent causal contribution or guaranteed loop penalties.
Illustrative example: In trajectories with the same terminal result, moving to high potential and returning to low potential can get different local credits instead of one shared value.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Why do local Transition and success inputs stay with the estimator rather than the actor?
GraphHCA: Closed-Form Hindsight Credit Assignment for Long-Horizon LLM Agents
Show answer and example
Answer: Terminal labels estimate training-side potential. The actor reads only the public state then available, not future success boundaries. No language model was trained or integrated into Evolve. Fixed-point convergence and credit rankings are mechanism evidence; full integration needs real rollouts, updates, and isolated evaluation.
Illustrative example: Labeling a terminal state after training interaction can help compute credit. Telling a test agent which successor leads to success leaks the result.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does Dr.Credit ask whether each deep-research retrieval adds new evidence for the answer?
Dr.Credit: Rubric-Grounded Process Credit Assignment for Deep Research Agents
Show answer and example
Answer: An agent can search repeatedly and collect near-duplicates without completing the facts needed. Dr.Credit links process feedback to rubric support so retrieval actually advances the answer. Rewards come from training-side judgments of support, not document counts; judge and rubric reliability still limit credit quality.
Illustrative example: Three reposts of a known price do not establish warranty terms. Finding the original warranty can add a missing answer requirement.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why shouldn't retrieving duplicate evidence repeatedly earn the same process reward?
Dr.Credit: Rubric-Grounded Process Credit Assignment for Deep Research Agents
Show answer and example
Answer: Credit should measure new support relative to accepted evidence, not tool calls. Track historical rubric coverage and score new or partial support to discourage reward farming. Retain outcome advantage so local novelty remains aligned with research goals, and require verifiable support relationships.
Illustrative example: Retrieving the same date twice earns no full second reward without a new constraint. Finding previously missing affiliation evidence adds support.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does incremental rubric-based process feedback differ in timing from grading only the final answer?
Dr.Credit: Rubric-Grounded Process Credit Assignment for Deep Research Agents
Show answer and example
Answer: Terminal scoring judges the deliverable but cannot easily isolate useful retrievals. Incremental support checks when individual requirements gain evidence and links feedback to those actions. Supported evidence can still be misused in the final answer; neither signal replaces the other, and evaluated actors must not see the training rubric.
Illustrative example: A retrieval can find the correct date and complete a process requirement while the final answer copies the year incorrectly. Terminal scoring should still mark that error.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
What can local fixture rubrics and support scores validate?
Dr.Credit: Rubric-Grounded Process Credit Assignment for Deep Research Agents
Show answer and example
Answer: They validate history updates, no repeat credit for duplicates, and direction of incremental credit. They do not test whether a semantic judge correctly recognizes support. The policy does not read gold answers, and a three-seed mini-suite is not Deep Research capability. Stronger claims need real sources, trustworthy matching, and actual training.
Illustrative example: A fixture stating that evidence A supports criterion one can test that the second retrieval adds zero credit. It does not test whether a model recognizes that support on a webpage.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why can't long multi-turn training trajectories simply be truncated by length?
Dependency-Aware Trajectory Refinement for Efficient Multi-Turn Agent Fine-Tuning
Show answer and example
Answer: Tool traces depend on observations, reasoning, and actions. Early or long content may support the final conclusion, and mechanical truncation makes later turns inexplicable. Dependency Refinement retains necessary semantic connections while reducing cost. Annotations may be wrong; shorter traces do not prove more learning or that deleted content never matters.
Illustrative example: If the first turn establishes authorized scope and the final action relies on it, keeping only recent messages makes the action appear unsupported.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why prune leaves first when compressing multi-turn traces instead of deleting the longest message?
Dependency-Aware Trajectory Refinement for Efficient Multi-Turn Agent Fine-Tuning
Show answer and example
Answer: A dependency DAG marks turns supporting the final answer; leaves are more likely to be redundant branches with no downstream dependency. Strict and then relaxed merging reduce repetition while preserving required information flow. The longest message may contain the key tool evidence, so length-only pruning can remove necessary causal context.
Illustrative example: A long log may contain the only error code, while short chat has no dependents. Prune the chat leaf rather than discarding the log because it is longer.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How do leaf pruning and turn merging differ in semantic preservation?
Dependency-Aware Trajectory Refinement for Efficient Multi-Turn Agent Fine-Tuning
Show answer and example
Answer: Pruning removes branches that no longer serve the objective. Merging compactly represents steps that can be expressed together. Strict merging applies tighter constraints; relaxed merging seeks more compression at greater risk. Preserve order and dependency invariants instead of substituting arbitrary summaries.
Illustrative example: Two reads of identical state can be merged. Keep the causal order between a change and verification, not a vague 'handled' summary.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Does the auditable local DAG include an actual LLM dependency annotator?
Dependency-Aware Trajectory Refinement for Efficient Multi-Turn Agent Fine-Tuning
Show answer and example
Answer: No. It prunes and merges a supplied fixture DAG to verify structure and rejection paths, not automatic semantic annotation. If input omits a key edge, pruning may legally drop a necessary turn. Compression ratios and mechanism passes are not agent fine-tuning quality or paper-level results.
Illustrative example: If a fixture omits 'the answer depends on turn two,' the pruner cannot infer the edge. Correct test graphs do not validate dependency extraction on open trajectories.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why might learning in one harness fail to improve performance under other interfaces?
Show answer and example
Answer: Tool protocols, prompts, and workflows change which states and actions are encountered. Some learning is specific to that organization. Harness Bandit allocates budget across harnesses for both learning and transfer. In-training gains are not cross-interface generalization; report evaluation conditions separately.
Illustrative example: A model familiar with one edit command may face different arguments elsewhere. More practice on the known interface can improve its score without improving transfer.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why schedule multi-harness training using both learnability and transferability?
Show answer and example
Answer: An easy harness may teach only its specific interface; a transferable but currently hard one may need exploration. Combining both signals and a bonus allocates limited budget and avoids overfitting to easy scores. Scheduling weights are training decisions; a reference function or multi-seed diagnostics are not cross-harness ability evidence.
Illustrative example: A provides many hints and B resembles deployment. Do not give A all budget for faster near-term gains without checking whether the learning transfers.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
What trade-offs distinguish uniform multi-harness allocation from bandit scheduling?
Show answer and example
Answer: Uniform allocation does not adapt to changing payoffs. A bandit can sample more from environments worth learning or uncertain at the current stage. It may favor short-term proxies and spend budget on exploration, so assess scheduling cost and transfer rather than only within-harness efficiency.
Illustrative example: After one harness saturates, shift budget to paths with transferable gains, while occasionally revisiting it to guard against stale estimates.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Do local online harness-selection counts establish completed multi-harness LLM RL?
Show answer and example
Answer: No. The reference runs gates, allocation, and state updates, without real LLM RL, full rollouts, or transfer benchmarks. Synthetic learnability and transferability check weight response, not portable capability. Three-seed scheduling statistics cannot establish training effects.
Illustrative example: More allocations to an arm with a supplied high transfer score verify the scheduler. A manually set score does not measure model transfer.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why must inference trajectories match optimizer examples when training large terminal agents?
T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks
Show answer and example
Answer: Terminal interactions include generated tokens, observations, and potentially changing execution paths. Reconstructed samples can optimize behavior that was never sampled. T1 keeps long rollouts traceably aligned with updates and supports scale. Record consistency is necessary for correct training, but does not establish capability or final gains.
Illustrative example: If tool output sits between model turns and concatenation changes token boundaries, computed probabilities may no longer represent the original action.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why train on token IDs actually sampled in rollouts rather than retokenizing complete interaction text?
T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks
Show answer and example
Answer: Joining tool observations and turns before tokenization can alter boundaries or token IDs, so policy probabilities no longer correspond to sampled actions. TITO retains sampled IDs, boundary repair handles drift, and MoE route replay addresses expert-path differences. Alignment is a training contract, not evidence that terminal tasks were learned.
Illustrative example: Concatenating a command and an observation can create different subtokens or include non-action content. Keep sampled IDs to align training with the original action.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does MoE route replay differ from replaying generated tokens alone?
T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks
Show answer and example
Answer: Expert routes may differ for identical tokens because of implementation or state. R3 saves and replays routing choices to narrow execution-path mismatch between training and inference. Token equality is necessary but insufficient; route equality also does not guarantee task gains without real model and log-probability comparisons.
Illustrative example: If a token originally selects experts A and B but training recomputes A and C, outputs and gradients can change despite identical token IDs.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Does a local exact-token/route audit cover training a 122B MoE?
T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks
Show answer and example
Answer: No. It checks sampled sequences and route replay contracts, without a 122B model, cloud sandboxes exceeding 300 turns, or full terminal RL. Log consistency does not demonstrate a smaller training–inference gap. Paper scores and gap measurements belong to the authors' real system, not fixtures with matching fields.
Illustrative example: A fixture route array that replays element by element confirms indexing, not actual MoE dispatch or backpropagation.
Sections in the Chinese source: 核心公式或操作, 本地复现, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why separate base-model capability from execution-interface effects in multi-harness RL?
What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents
Show answer and example
Answer: Different prompts, tools, and workflows change success on the same task; grouping changes the meaning of rewards. Multi-Harness RL examines their effects on learning and transfer. If the base, tasks, or samples also change, attribution is difficult. Control trajectories and evaluation definitions instead of calling all cross-harness gains capability growth.
Illustrative example: One harness fills arguments automatically while another requires the model to fill them. Success differences may reflect interface help rather than base reasoning.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
What confound is isolated by freezing task–harness trajectories and changing only advantage grouping?
What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents
Show answer and example
Answer: Multi-harness training changes both data diversity and credit comparison groups. Comparing complete systems confounds them. Fixed trajectories let within-task comparison of cross-harness and within-harness normalization more directly test grouping. Report transfer differences and uncertainty on unseen minimal harnesses.
Illustrative example: Reuse trajectories for one repair task under two interfaces and recompute grouped advantages without resampling. This reduces attribution of sampling changes to the credit rule.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How do Within and Cross change the relative meaning of the same reward?
What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents
Show answer and example
Answer: Within normalizes inside each task–harness group, comparing sibling trajectories in one environment. Cross normalizes across harnesses for the same task, including between-environment reward differences in advantages. It can learn environment preferences; portability needs paired tests on an unseen minimal harness, not just training means.
Illustrative example: A stronger-tool harness can earn generally positive Cross advantages because its rewards are higher. That does not imply better code repair on a minimal interface.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Why does a held-out Cross−Within interval spanning zero fail to establish transfer?
What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents
Show answer and example
Answer: A small positive point estimate is still compatible with no change; interpret the paper alongside its paired confidence interval. Old local gold-reading scores are invalid. Current checks cover grouping, gradients, and protocols, without Qwen RL, actual coding harnesses, or a sealed oracle. Counts are not SWE-bench capability.
Illustrative example: A 0.25 percentage-point gain with negative values inside the interval does not establish reliable superiority over Within. Do not omit the interval and report only the positive estimate.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does ACE say more agent trajectories do not necessarily make better training data?
What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents
Show answer and example
Answer: Trajectories may be incorrect, too hard for the learner, or repetitive, and more data can amplify those issues. ACE selects for Accuracy, Complexity, and Diversity to account for quality and learner fit. Define and evaluate each empirically rather than inferring benefits from more steps or topics.
Illustrative example: A thousand copies of an incorrect tool call are not good data. Many simple tasks the learner has mastered may also teach little.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why must Complexity be calibrated to a specific learner instead of assigning a universal difficulty label?
What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents
Show answer and example
Answer: Learnability differs across models. Very easy examples add little signal and very hard ones may only generate failures. Accuracy first ensures credible samples, learner-relative complexity finds a useful band, and Diversity adds environment and behavior coverage. All three matter; length and tool count do not establish better data.
Illustrative example: A novice can learn from three-step tool tasks and fail every twenty-step one. A stronger model may need the latter, so the curriculum band should depend on the learner.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
Why keep Complexity as learner challenge and Diversity as coverage separate?
What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents
Show answer and example
Answer: Complexity measures difficulty for the specified learner; Diversity measures differences among samples and coverage of task space. Diverse data can be easy and difficult data can repeat one pattern. Both require Accuracy: a novel but wrong trajectory is not good supervision, and static labels do not establish training benefit.
Illustrative example: Ten different tools used once each offer broad but easy coverage. Ten similar hard error cases challenge the learner but cover a narrow area.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
How does public artifact evaluation prevent three-stage admission from becoming fictitious training results?
What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents
Show answer and example
Answer: Pin an actual dataset revision; record environments, questions, trajectories, verifier, and learner loss. Compare random and three-stage selection under equal budgets. The local path has admission contracts and artifact replay, not trained checkpoints or SOTA claims. Do not fill missing provenance or failures with synthetic high scores.
Illustrative example: A JSONL export that passes field checks is an auditable input. Without training and independent model evaluation, it cannot show an ACE success-rate gain.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why can asynchronous agent RL depend on which task finishes first?
Show answer and example
Answer: Long tasks have variable durations, so asynchronous training uses results as they arrive. Completion order can correlate with difficulty and policy version. SPO++ keeps throughput while aligning advantage baselines and statistics with the correct generation events. Fast completion is not representativeness; track sampling-time conditions rather than backfilling from later outcomes.
Illustrative example: Easy tasks may return early and change historical values. A hard task dispatched earlier should not be treated as sampled with that new baseline.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why weight advantage standardization by action-token count for a token-mean actor loss?
Show answer and example
Answer: Equal trajectory votes during whitening conflict with token-mean loss and change how long and short rollouts influence updates. Token weighting aligns normalization with the optimization measure. Freezing the baseline at dispatch prevents completion-order leakage; it is separate from redefining rewards by text length.
Illustrative example: A 100-token trace occupies more loss positions than a 10-token trace. Giving them one vote each in normalization would mismatch actual gradient averaging.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
What changes between freezing a baseline at dispatch and using the latest baseline at completion?
Show answer and example
Answer: Freezing associates each rollout with prompt value visible when dispatched. The newest value at completion incorporates later arrivals, affecting rollouts differently by duration. SPO++ indexes evidence by generation-policy events, so later history cannot override frozen conditions. This correction is independent of token-weighted standardization.
Illustrative example: A goes out before B but finishes after it. B can update future baselines without retroactively entering A's dispatch-time history.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Do local event-time and token-measure checks include asynchronous Qwen training?
Show answer and example
Answer: No. A deterministic mini-suite checks frozen values, history timing, and action-token weighting, without real asynchronous models, ALFWorld, or Math-TIR. Event counts verify branches, not paper reward-curve area or throughput gains.
Illustrative example: Out-of-order completion of ten fixtures tests timing, not real worker latency, policy staleness, or parameter updates.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does AHEAD want more than one success or failure signal across an entire long-horizon agent trajectory?
AHEAD: Agentic Hints for Effective Agent Development
Show answer and example
Answer: A few critical errors may determine the outcome, while other steps are reasonable or merely affected by them. Uniform credit cannot identify what to fix. AHEAD uses targeted teacher signals to adjust learning on those steps. Fine-grained credit depends on error localization, and privileged teacher information is not available to the deployed agent.
Illustrative example: If only the third retrieval query strays, later failure should not assign equal responsibility to every correct evidence collection step.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why give a corrective hint only to the teacher on critical error steps instead of sharing full answers at every step?
AHEAD: Agentic Hints for Effective Agent Development
Show answer and example
Answer: Ordinary steps already get environment feedback. Blanket privileged hints waste signal and hide where correction matters. After locating an error, compare log probabilities for the same generated action with and without the hint, then adjust terminal advantage within bounds. Hints stay on the training teacher side and cannot enter deployment student context.
Illustrative example: If only the third step chooses the wrong tool, target that choice. Letting every step see the final solution is not autonomous completion.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does multiplying the original episode advantage differ from using the teacher gap as a new reward?
AHEAD: Agentic Hints for Effective Agent Development
Show answer and example
Answer: Multiplication modulates the original advantage within clip and λ limits, preserving the terminal signal. Replacement would turn teacher preferences into a separate objective. The gap is a likelihood difference for the same token in two contexts, not success probability or causal contribution.
Illustrative example: A hint's stronger support for a token in a failed trajectory does not unconditionally reward it as success; original advantage and the bounded factor still apply.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
How do deterministic local error locations, real ToolRoute, and the paper's LLM analyzer differ?
AHEAD: Agentic Hints for Effective Agent Development
Show answer and example
Answer: The mini-suite uses deterministic positions to test hint routing, with no LLM analyzer, Qwen GRPO, or three original benchmarks. ToolRoute is a different isolated feedback protocol, not complete algorithm training. Hints, gold, and reference actions stay in permitted training contexts and are not visible to the test policy.
Illustrative example: A fixture that identifies step six as wrong tests gating, not autonomous localization of that step in a real long failure.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does AutoSaddler turn one task's failures into workflow improvements usable later?
AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces
Show answer and example
Answer: Prompt, tool configuration, and middleware flaws can repeatedly cause long-task failures. Fixing one answer does not address shared causes. AutoSaddler proposes bounded harness edits from execution evidence and maintains persistent versions. Only execution validation and later reuse show a patch's effect; version counts are not capability.
Illustrative example: Repeatedly skipping pre-submission tests suggests a shared workflow change. Manually testing the current task fixes one deliverable, not future omissions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why select patches on a development set and record EvoDAG after a same-batch check?
AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces
Show answer and example
Answer: The original failures help diagnose direction but can be overfit by a targeted patch. Recompare on development data; the version graph records parent harness and evidence for future iteration and rollback. Bounded structured patches do not imply semantic safety, and final reporting tests cannot choose updates.
Illustrative example: After fixing three identical errors, check other development tasks for regressions and save the parent–child relation instead of overwriting the only harness file.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
Which persistent state changes when editing harness code versus base-model weights?
AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces
Show answer and example
Answer: Harness edits change how prompts, tool configuration, or middleware organize later execution. Weight updates change the model's behavior distribution. AutoSaddler iterates on the harness as a persistent code asset; workflow repair is not model training. The approaches can complement one another but require separate evidence, versions, and rollback targets.
Illustrative example: A recovery branch after failed tests changes workflow code. If the checkpoint is unchanged, its parameters have not learned error recovery.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
What do local durable-update and reuse counts establish?
AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces
Show answer and example
Answer: They show that the optimization loop, validation gate, and persistent reuse state run on a public mini-suite, without original GAIA2, SWE-Bench Pro, Terminal-Bench, or production harnesses. Diagnostic and write counts are not real LLM diagnosis, and proxy cost is not API cost, human efficiency, or online benefit.
Illustrative example: Twelve rule updates and 108 reuses verify the version path. Without real models and tools, the authors' nine-to-ten-point gain does not carry over.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does Agent-G² provide partial expert trajectories before exploration rather than relying only on random trial and error?
Agent-G²: Gaussian Guidance for Agentic Reinforcement Learning
Show answer and example
Answer: An expert prefix brings the policy to a state with more learning value, then the current policy explores from there. Existing rollouts estimate guidance proportions by difficulty cluster, balancing help with autonomy and avoiding costly per-example probes.
Illustrative example: A hard task can receive a longer start, while the policy takes over earlier on an easy task.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why sample guidance depth by difficulty cluster rather than retain the same expert-prefix length for every task?
Agent-G²: Gaussian Guidance for Agentic Reinforcement Learning
Show answer and example
Answer: Too much prefix on easy tasks reduces independent exploration; too little on difficult tasks can produce only failures. Cluster success rates and variability from existing rollouts set the center and variance of a guidance band, then depths are sampled per task, avoiding extra per-item probes. Training prefixes are scaffolding and must not be provided in independent evaluation.
Illustrative example: An easy door task may need a very short hint; a complex multiroom task can need a longer start. Giving both half a prefix can overguide one and underguide the other.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does cluster-level guidance-band estimation differ in cost from repeated per-task probes?
Agent-G²: Gaussian Guidance for Agentic Reinforcement Learning
Show answer and example
Answer: Clusters reuse success rates and variation from existing policy rollouts, reducing extra per-task trials. Individual probes can be finer but cost additional rollouts. Neither fixed nor cluster-based help establishes optimal depth for each task.
Illustrative example: Similar tasks can share a depth distribution rather than repeatedly testing every task to find a starting point.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
What does Agent-G² do if it samples a negative guidance depth or a proportion beyond the expert trajectory?
Agent-G²: Gaussian Guidance for Agentic Reinforcement Learning
Show answer and example
Answer: Bound the mean and clip Gaussian samples to [0,1], with a variance floor. Expert prefixes are training guidance; the hidden expert plan must not be supplied to policies in independent capability evaluation.
Illustrative example: Clip r=1.2 to 1 rather than indexing past the trajectory or treating overflow as extra oracle information.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does AUSO both internalize skills and retain an action-level skill utilization stage?
AUSO: Action-Level Unified Skill Optimization from Internalization to Utilization
Show answer and example
Answer: Retrieving external skills costs context, but fully internalized skills are harder to select per task. AUSO first uses skill-conditioned teachers for general internalization, then explores and compares action distributions with and without skills to adjust learning emphasis.
Illustrative example: Common steps can be learned into the policy while a complex action may still benefit from a particular skill hint.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why bound action-level JSD weighting and preserve the original trajectory advantage's sign?
AUSO: Action-Level Unified Skill Optimization from Internalization to Utilization
Show answer and example
Answer: Distribution differences with and without skills show action sensitivity, not whether the action benefited. Preserve the outcome-advantage sign so local weights only adjust credit magnitude and a highly divergent step in a failed trace does not become positive reward. Separate internalization, exploration, and utilization to reduce persistent external skill-context costs.
Illustrative example: If an action in a failed run heavily depends on a skill, it can receive stronger corrective learning, not a success reward merely for depending on that skill.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
Do AUSO's action-level JSD and whole-trajectory success tell us the same thing?
AUSO: Action-Level Unified Skill Optimization from Internalization to Utilization
Show answer and example
Answer: Success describes the outcome of the whole process. Action-level differences show a particular step's sensitivity to skill conditioning. A large JSD does not establish causal benefit; combine it with outcomes and bounded weighting rather than equating divergence with improvement.
Illustrative example: Only one tool choice in a successful trajectory may really be affected by the skill.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Why compare skill-conditioned and unconditioned policies at the same h_t and action-distribution granularity?
AUSO: Action-Level Unified Skill Optimization from Internalization to Utilization
Show answer and example
Answer: Changing other context confounds JSD. Compare the distributions at the same action boundary, and verify that actual w is nonnegative and bounded. Writing 1±β alone does not ensure the advantage sign is preserved.
Illustrative example: If one distribution sees a tool result and the other does not, the difference cannot be attributed only to skill S.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does SAPO use one autoregressive backbone for policy and value?
SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning
Show answer and example
Answer: A shared backbone reads action decisions and value signals at different causal positions and combines PPO, on-policy SARSA, and trajectory GAE. Sharing parameters does not erase the different input boundaries.
Illustrative example: Policy information available before action generation and later information permitted for value estimates must follow the protocol separately.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why specify different causal boundaries when one autoregressive backbone outputs both policy and value?
SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning
Show answer and example
Answer: Action probabilities use only visible history before the decision, and value estimates must match the correct time. Post-action observations can leak future information into advantages. A shared backbone saves structure, but PPO, SARSA, and GAE still need state/action alignment; a single rollout does not remove bootstrap and termination semantics.
Illustrative example: The value before a command cannot already see the command's passing test result. Such credit would be precise only because the future leaked.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does single-rollout optimization differ from sampling many branches at one state for group comparisons?
SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning
Show answer and example
Answer: SAPO organizes policy, value, and temporal advantage from a single on-policy trajectory. Branch-group comparison needs multiple rollouts. Data requirements, value estimation, and credit assignment differ, regardless of whether final success rates match.
Illustrative example: An executed trajectory does not give the actual returns of untried alternatives at the same state.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Why retain trajectory boundaries when calculating GAE instead of concatenating episodes?
SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning
Show answer and example
Answer: Advantages accumulate later TD residuals. Concatenating episodes would propagate another task's rewards into the current one. Respect terminal boundaries for bootstrapping and accumulation. The deterministic local mini-suite is not a browser run or full RL training.
Illustrative example: The first record after task A ends belongs to B and is not A's future reward.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does SPADE use the same LLM as environment designer and inference agent?
SPADE: Self-Play in Adaptive Synthetic Executable Environments
Show answer and example
Answer: The designer creates executable synthetic environments, the agent tries them, and the difference between hinted and unhinted rewards identifies frontier tasks. The aim is learnable challenges rather than arbitrary unsolvable problems.
Illustrative example: An environment that fails without hints but succeeds with them may be more informative than one that fails both conditions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why use hinted versus unhinted regret for the environment designer rather than simply maximizing agent failure?
SPADE: Self-Play in Adaptive Synthetic Executable Environments
Show answer and example
Answer: Failure alone can reward impossible or overly hard environments. A hint that substantially improves a difficult unhinted outcome identifies a more learnable frontier. Environments must still execute, and privileged information stays in training comparisons rather than leaking into unhinted evaluation.
Illustrative example: A difficult maze solvable with direction hints can be useful. A maze with no exit fails both agents and should not score well just for being hard.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
Do maximizing failure rate and maximizing hint/no-hint regret favor the same environments?
SPADE: Self-Play in Adaptive Synthetic Executable Environments
Show answer and example
Answer: No. Failure alone can favor unsolvable or meaningless environments. Regret also requires privileged help to improve return, distinguishing solvable but currently difficult tasks. It still needs real execution and feedback rather than a fixed difficulty label.
Illustrative example: An environment that scores zero for both agents has a high failure rate but zero regret.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
May a SPADE agent retain hints in the no-hint evaluation condition?
SPADE: Self-Play in Adaptive Synthetic Executable Environments
Show answer and example
Answer: No. Isolate condition inputs. Privileged hints belong only to their training or comparison branch and must not enter the unhinted policy. Executable environment checks still matter; the local mini-suite did not run original external environments or large-model training.
Illustrative example: If an unhinted callback still receives the designer's saved plan, the return difference cannot be interpreted.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does RTPO organize multi-turn decisions into reverse updates?
RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training
Show answer and example
Answer: Early action value depends on downstream continuation. RTPO arranges rollouts in a sparse reverse tree and updates turns backward to keep decisions aligned with the downstream policy evaluating them.
Illustrative example: Handle the consequences of late actions first, then return to earlier decisions rather than treating the trajectory as unstructured text.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why do reverse-turn updates help keep decisions aligned with on-policy downstream continuations?
RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training
Show answer and example
Answer: Early action value depends on later policies. If downstream behavior has changed substantially, credit from old continuations can be inconsistent. A reverse tree handles turns near the outcome first, then propagates evidence backward so early decisions reflect the updated continuation. Update order does not remove importance or execution-alignment checks.
Illustrative example: After the last two steps learn recovery, assess opening decisions using that new recovery ability instead of values from the old failing continuation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
Are reverse-turn updates the same as training on reversed token text?
RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training
Show answer and example
Answer: No. Reverse order applies to decision updates and continuation dependencies. Causal language-model input still follows the legal trajectory. Reversing text changes the task and visibility and cannot substitute for reverse policy optimization.
Illustrative example: Updating turn three before turn two does not mean reading turn-three observations to predict a turn-one action.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Why record both behavior policy and downstream continuation version for each RTPO turn?
RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training
Show answer and example
Answer: Early advantages based on mismatched later policies can lose on-policy correspondence. Reversing indices alone is insufficient: check tree nodes, state, and continuation ownership. Local control flow is not complete external-environment RL.
Illustrative example: After a node is reevaluated under a new continuation policy, its old continuation reward is not the current return for the same state.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does PlanPO split planning efficiency into turn count and response length per turn?
PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs
Show answer and example
Answer: Fewer environment interactions and shorter generated responses are distinct costs. PlanPO compares successes coarse-to-fine and combines efficiency with outcome advantage, retaining completion while improving the process.
Illustrative example: Two successful traces can differ in tool calls and explanation length; evaluate those dimensions separately.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why compare turn counts and response lengths only among successful traces rather than rewarding every short trace?
PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs
Show answer and example
Answer: Short can mean efficient planning or premature abandonment. First separate successes by outcome, then compare turns and response length among comparable successes. Coarse-to-fine advantages supplement outcome, not correctness, and fewer tokens cannot excuse skipped evidence collection.
Illustrative example: Two successful bug fixes can differ by three wasted searches. A shorter trace that declares completion without editing files should get no efficiency reward.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does PlanPO efficiency among successes differ from rewarding the shortest answer?
PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs
Show answer and example
Answer: A direct length reward can favor failure or early stopping. Comparing successful traces first requires completion, then optimizes turns and length. Length advantage is not global outcome advantage and must remain separate.
Illustrative example: An unexecuted one-line 'done' should not beat a successful multi-turn execution just because it is short.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Why distinguish agent responses from environment observations when counting turns and response lengths?
PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs
Show answer and example
Answer: The two axes measure interaction turns and per-turn generation. Counting long tool returns as agent output changes credit. Record success filtering, counting boundaries, and each advantage coefficient, not just total mean tokens.
Illustrative example: A tool can return a long page without the agent generating the same number of tokens.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does TRCA define a rubric for every state transition rather than scoring only the outcome?
TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agents
Show answer and example
Answer: Terminal outcomes hide important actions. Transition rubrics separately check evidence, execution, and invalid behavior and add breakthrough rewards. They do not require a successful anchor and supply more detailed credit.
Illustrative example: A step can gather key evidence while failing execution, with separate scores rather than zeroing both.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why score Evidence, Execution, and Invalidity separately instead of one process score?
TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agents
Show answer and example
Answer: A transition can add evidence but fail to execute, or be well formed without advancing the task. Rubric components distinguish those cases and penalize invalid behavior. Breakthrough credit identifies key progress without a successful anchor, but still needs actual transitions, not fixed template scores.
Illustrative example: A search can reveal an error cause, followed by a command syntax failure. A total score alone obscures whether retrieval or invocation needs fixing.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does TRCA's lack of a success anchor differ from feedback that needs a successful reference trace?
TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agents
Show answer and example
Answer: Reference-based approaches first need a comparable successful execution. TRCA derives base and breakthrough rewards from actual transition rubrics without first finding a success. This is a feedback prerequisite, not proof of a higher task-success rate.
Illustrative example: Even when both full traces fail, a transition can collect useful evidence and receive rubric feedback.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Why verify the sign and component logs when aggregating TRCA invalidity?
TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agents
Show answer and example
Answer: Subtract r_invalid and add the other base and breakthrough terms as defined. Missing the negative sign rewards invalid behavior. Keep component scores linked to transitions, since a total can conceal mistakes. Fixed values are not real evaluation.
Illustrative example: With other terms fixed, increasing invalidity must lower total reward.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does SinkFlex-RL address both rollout environments and attention memory?
Efficient Reinforcement Learning for Long-Horizon Tool-Use Agentic Tasks
Show answer and example
Answer: Long tool traces require multi-turn state management and face context and training-memory bottlenecks. It combines environments, fresh trajectory groups, value-model-free GRPO, and sink-aware FlexAttention, not just a compressed mask.
Illustrative example: Saving attention memory still requires correct environment state for actual multi-turn policy updates.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why preserve model-specific sink scaling when using a sliding-window or causal mask?
Efficient Reinforcement Learning for Long-Horizon Tool-Use Agentic Tasks
Show answer and example
Answer: Sparse visibility and sink scaling affect different parts of attention. Mask changes do not justify dropping the model's scaling. Training still needs real environment state, long context, and GRPO data flow; a passing formula or kernel is not full on-policy rollout and tool-task verification.
Illustrative example: If the window retains recent text and sink but also removes special sink scaling, output changes combine two interventions and cannot be attributed to the window alone.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How do GRPO group-relative advantage and PPO with an independent value model differ in training dependencies?
Efficient Reinforcement Learning for Long-Horizon Tool-Use Agentic Tasks
Show answer and example
Answer: This approach builds relative advantages from groups without a separate value model, then applies a clipped policy objective and KL regularization. Changing advantage estimation alters rollout and memory needs; it is not an optimizer name change.
Illustrative example: Grouped returns provide a relative baseline, not a trained per-state critic.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Why not replace SinkFlex-RL token averaging with per-trajectory averaging and equal weights?
Efficient Reinforcement Learning for Long-Horizon Tool-Use Agentic Tasks
Show answer and example
Answer: The denominator is total valid length ΣT_i, giving long and short trajectories a different weighting from equal per-trajectory means. Preserve model sink scaling and verify the actual mask. The local mini-suite does not reproduce FlexAttention GPU memory curves.
Illustrative example: Trajectories of lengths 10 and 100 contribute differently to the token mean than to a mean with half the weight each.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does ResidencyRL use multi-turn simulation instead of only static knowledge questions?
ResidencyRL: Reinforcement Learning in Simulated Clinical Environments
Show answer and example
Answer: Active inquiry, tool use, and long-horizon decisions need observed interaction effects. The method connects scenario generation, information-asymmetric environments, isolated multidimensional judges, and GRPO to train an interactive policy rather than memorizing answers.
Illustrative example: The agent must decide what to ask next from new observations rather than receiving all hidden scenario information up front.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why keep hidden references on the environment and judge side in simulations with asymmetric information?
ResidencyRL: Reinforcement Learning in Simulated Clinical Environments
Show answer and example
Answer: The policy must gather evidence through questions, tools, and observations. Direct reference access bypasses long-horizon decision making. Isolate scoring dimensions before aggregation to reduce contamination, but the reward formula is not the full environment or professional capability. Local diagnostics are not clinical efficacy or decision reliability.
Illustrative example: If a state should be revealed only after querying it, do not provide the reference summary in the policy prompt and call the result autonomous evidence collection.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does aggregating six reward dimensions differ from implementing the full simulation training algorithm?
ResidencyRL: Reinforcement Learning in Simulated Clinical Environments
Show answer and example
Answer: Aggregation combines already obtained normalized scores and penalties. The method also requires scenarios, actual interactions, separate judges, and policy updates. Fixed scores or a loss wrapper cannot replace that chain.
Illustrative example: Combining six numbers into R does not establish that the model performed and learned from a simulated interaction.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Why cap the ResidencyRL penalty with min(3,P) rather than arbitrarily clipping total reward?
ResidencyRL: Reinforcement Learning in Simulated Clinical Environments
Show answer and example
Answer: First calculate three times the weighted normalized dimension scores, then subtract a penalty capped at three. Preserve order and the distinct terms. Hidden scenario information is for the environment and judges; isolated judge calls must not pass the reference to the policy.
Illustrative example: Increasing P from 4 to 10 still deducts 3; it does not change the other dimensions' scores.
Sections in the Chinese source: 核心公式, 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does EnvACE switch the same policy into a rehearsal role?
Show answer and example
Answer: It predicts the next observation between real actions, performing limited private look-ahead without a separately trained world model. Training constrains acting and rehearsal; a planning prediction is not a tool execution.
Illustrative example: The agent imagines a likely response before deciding whether to take the action.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why compute group-relative advantages separately for acting and rehearsal?
Show answer and example
Answer: Action success and observation-prediction rewards have different scales and objectives. Centering them together lets one role's high scores suppress the other's learning. Separate groups let the shared policy learn environment prediction and plan with private rehearsal. Imagined observations are not tool feedback or execution evidence.
Illustrative example: A high score for predicting an open door should not mistakenly reward a failed real action; keep rehearsal distinct from the actual tool result.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does same-policy rehearsal differ architecturally from a separate world model?
Show answer and example
Answer: Rehearsal reuses the agent policy under a role condition. An independent world model has its own prediction model and training and maintenance costs. Reuse does not guarantee accuracy: keep imagined observations and real feedback separate and compute role advantages separately.
Illustrative example: 'The tool may succeed' from rehearsal is not a successful tool return.
Sections in the Chinese source: 背景与主要改动, 本地复现. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Why exclude rehearsal predictions from EnvACE's real tool-response count?
Show answer and example
Answer: Counting them would disguise imagination as execution evidence. The documented mini-suite has real_tool_responses=0; rehearsal and role-wise updates establish those paths, not browser or real-tool success.
Illustrative example: Three predicted observations are three rehearsals, not three actual API calls.
Sections in the Chinese source: 本地复现, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does CIPO add evidence credit to intermediate search steps?
Contextual Information Policy Optimization for Search Agents
Show answer and example
Answer: Final answer reward does not show whether retrieval affected decisions and may encourage confirming prior beliefs. CIPO credits later action changes informed by external evidence and combines dense credit with global outcomes.
Illustrative example: Revising a plan after finding counterevidence is more informative than appending an irrelevant citation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why can terminal answer rewards foster confirmation bias, and how does dense evidence credit help?
Contextual Information Policy Optimization for Search Agents
Show answer and example
Answer: The model may guess first and retrieve only supporting information, with an accidentally correct answer hiding poor search. Credit turns where external evidence changes later actions to reward use of information rather than call counts. Keep global outcomes so novelty does not divert attention from the question.
Illustrative example: If a search disproves a guessed date and the agent checks another source, distinguish that evidence-driven change from repeating the original guess.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does CIPO evidence-impact credit differ from rewarding citation counts?
Contextual Information Policy Optimization for Search Agents
Show answer and example
Answer: Citation counts measure surface form. Evidence credit concerns whether external information affects later actions. Seeing retrieved text alone does not establish causal influence; verify it under the method's definition.
Illustrative example: Ten links that change nothing may be less useful than one piece of evidence that changes a decision.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Which advantage remains at the λ=0 and λ=1 boundaries in CIPO?
Contextual Information Policy Optimization for Search Agents
Show answer and example
Answer: λ=0 uses only outcome and λ=1 only evidence, checking the mixing direction. Intermediate values combine them. Bind turn credit to real trajectory positions, and do not replace local mini-suite results with the paper's large-model scores.
Illustrative example: If evidence still changes updates at λ=0, check coefficients or a leftover branch.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does AgentOPSD use turn evidence to update success belief recursively?
Show answer and example
Answer: Trajectory rewards do not identify key decisions. Aggregate teacher/student token log-probability gaps from privileged replay by turn, accumulate them in log odds, and assign local credit from successive belief changes.
Illustrative example: A tool choice that sharply raises belief is easier to localize from the change than from an already high level.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why identify pivotal turns from changes between adjacent success beliefs rather than the final belief?
Show answer and example
Answer: The final value mixes all prior evidence and cannot localize the decisive turn. Accumulate teacher–student token probability differences in log odds and use the per-turn sigmoid belief increment to identify a local change. This is hindsight attribution for training; privileged replay and references stay out of independent evaluation policies.
Illustrative example: If a tool response causes belief to jump after several uneventful steps, its increment identifies the turn better than broadcasting a high final belief.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
Does critic-free AgentOPSD require no teacher information?
Show answer and example
Answer: No. It has no independent value critic, but turn evidence still uses teacher/student differences in privileged replay. Mini-suite state updates do not establish actual large-model teacher rollouts and policy training.
Illustrative example: No critic network does not remove the teacher distribution required for e_t.
Sections in the Chinese source: 背景与主要改动, 本地复现. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Why accumulate AgentOPSD evidence in log odds and take sigmoid differences?
Show answer and example
Answer: Update log-odds state ℓ_t=ℓ_{t−1}+e_t and credit the difference between adjacent sigmoid beliefs. Adding e_t directly to a probability can leave its range and changes accumulation. Compute large log odds stably, and keep privileged replay out of evaluation policy inputs.
Illustrative example: The same e_t changes different prior beliefs by different amounts; it is not a fixed additive probability adjustment.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
How does State2State construct training tasks automatically through environment exploration?
State2State: Environment-Derived Mid-Training for LLM Agents
Show answer and example
Answer: Exploration yields reachable initial and target states. A fixed projection T matches terminal state as the verifier, giving tasks without human-written descriptions or expert trajectories for every item. The policy still finds its own path.
Illustrative example: Knowing a room's initial and target layouts does not identify the sequence of actions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why can tasks built from initial and target states omit expert action trajectories?
State2State: Environment-Derived Mid-Training for LLM Agents
Show answer and example
Answer: The goal says where to arrive while the policy explores how. Rule-based state matching only verifies the endpoint, separating task generation from demonstrations and widening mid-training sources. Verifier-internal goals or rules are not stepwise correct-action hints. Matching must preserve task-relevant state.
Illustrative example: Given initial and target object layouts, the agent moves objects itself and the environment compares the result; it does not need a prescribed left-then-right sequence.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
Does target-state supervision give the same information as behavior cloning from an expert?
State2State: Environment-Derived Mid-Training for LLM Agents
Show answer and example
Answer: A target state defines success; behavior cloning also provides the action sequence. Goal supervision permits multiple valid routes. Passing verifier-computed expert paths to the policy would invalidate the no-demonstration setting.
Illustrative example: Two different paths can both succeed by reaching the goal, without copying a canonical sequence.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Why predefine the State2State projection in T(s_T)=T(s*)?
State2State: Environment-Derived Mid-Training for LLM Agents
Show answer and example
Answer: The projection determines which differences matter. Changing T after observing results changes the task. Verify initial and terminal states, legal transitions, and matching fields. A local state operator does not replace actual large-model mid-training in the paper's environment.
Illustrative example: If a goal specifies object positions, do not remove position from matching after a failure to manufacture success.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does OCSD worry that replay scaffolding itself changes token scores?
Show answer and example
Answer: Extra structure and prompt formatting during future-observation replay can affect predictions independently of the observation. Use structurally matched Full and Observation-Ablated controls to extract a residual better focused on the observation.
Illustrative example: A long replay template can change probabilities, so not every change comes from environmental evidence.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why subtract structurally matched Observation-Ablated replay from Full replay?
Show answer and example
Answer: Comparing replay with future observations directly against the student conflates observation content and extra scaffolding. Keep the structure and remove only observations, then subtract to isolate information contribution better and amplify updates only on uncertain steps. Future information is for training attribution, not test-time input.
Illustrative example: If the replay template adds 'please check,' keep it in both versions. Only observation availability should differ so the prompt effect is not counted as observation evidence.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
What does the Full–Ablated residual control for beyond the Full–Student gap?
Show answer and example
Answer: A single gap includes both observation content and scaffolding. A matched ablated condition removes the observation while preserving structure, so subtraction cancels shared effects. Unmatched templates leave new confounds.
Illustrative example: If Full alone includes another instruction, Full−Ablated also captures that instruction's effect.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
How do the two student log probabilities cancel in the OCSD residual?
Show answer and example
Answer: For the same token and student condition, r=(logπ_full−logπ_S)−(logπ_abl−logπ_S)=logπ_full−logπ_abl. If the student values come from different prefixes or alignments, fix matching before cancellation.
Illustrative example: Compare both expressions on the same input to detect sign or position mistakes.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does EvoHarness-RL make runtime-state reads and writes trainable actions?
EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents
Show answer and example
Answer: If fixed scripts maintain Belief, Progress, and Experience, the policy cannot learn when updates or merging are worthwhile. SFT teaches harness actions, then task rewards and costs train when to use them.
Illustrative example: A simple task may need few experience edits, while a long task benefits from saving intermediate progress.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why make Belief, Progress, and Experience external state the policy can read and write?
EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents
Show answer and example
Answer: A fixed harness can make inappropriate decisions about storage and merging. Explicit actions teach the policy to manage belief, progress, and experience, using SFT to learn the interface and then optimizing task reward minus management cost. More writing is not inherently useful, and state must reflect actual observations.
Illustrative example: Write new constraints to Belief and merge duplicates into Experience. Appending every log each turn can consume more than the task gains.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does a learned harness policy differ from configuring fixed memory?
EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents
Show answer and example
Answer: A fixed module prescribes read/write behavior. A learned harness policy chooses when to read, write, or merge and pays operation costs. An 'evolvable' registry label on a fixed pipeline does not establish learned external-state management.
Illustrative example: Saving all history every turn is different from deciding whether to save based on current progress.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Why bind EvoHarness-RL's harness cost to actual execution logs?
EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents
Show answer and example
Answer: In R=R_task−λC_harness, cost must reflect executed reads, writes, and merges, not proposals alone. Check whether later policy steps read the updated state. The local mini-suite is not full paper-scale LLM RL.
Illustrative example: Text saying 'merge experience' without a state update is not a successful harness action.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does HindSearch generate hindsight critique after a failed search trace completes?
HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learning
Show answer and example
Answer: Terminal success is sparse. A frozen judge can use gold to identify problems in a completed failure and supply auxiliary on-policy distillation signals. Hindsight explains training failures; it is not future answer information available during evaluation.
Illustrative example: After training execution, note that search ignored key evidence to improve the next policy rather than answer the current test item.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
If a judge can read gold to generate critiques, why can't that critique be supplied directly during search evaluation?
HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learning
Show answer and example
Answer: Gold helps the training judge explain failures and turn sparse outcomes into actionable distillation signals. A reference-derived critique for the same evaluation task indirectly discloses answer information, invalidating independent search. Freezing the judge does not remove privileges in its inputs.
Illustrative example: Training feedback can use the correct date to identify a wrong source, but testing must provide only the question, not a critique revealing that Wednesday is correct.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
What does a trajectory-level hindsight critique add beyond binary success or failure values?
HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learning
Show answer and example
Answer: Critique can identify specific shortcomings and their relationships, while binary outcomes only show success. A teacher or judge produces it at a cost and may be wrong; longer explanations are not automatically more accurate supervision.
Illustrative example: Two zero-scoring traces could reflect missing retrieval versus misread evidence, which a single zero does not distinguish.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Who can read gold in HindSearch, and which inputs must stay out of independent search evaluation?
HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learning
Show answer and example
Answer: Only the frozen training judge and independent scoring side get gold. Critique derived from it is privileged training feedback. The evaluated policy cannot receive the current item's gold, judge object, or gold-derived hints without contaminating success rates.
Illustrative example: Placing a judge's answer-based corrective suggestion in a test search prompt is not leakage-free inference.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does TAPO reuse observed environment transitions to add training signal?
Show answer and example
Answer: A rollout contains each post-action observation as well as terminal reward. Predicting s_{t+1} from (s_t,a_t) teaches the shared backbone action consequences without additional expert demonstrations.
Illustrative example: Use actual tool responses to teach the model what feedback its actions produce.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why can learning next observations from reused rollouts add training signal without adding deployment inference steps?
Show answer and example
Answer: Each transition already has state, action, and next observation. An auxiliary prediction target on the shared backbone uses them without new expert or policy rollouts. Deployment keeps the original policy without explicitly calling a predictor. Controlled comparisons must establish task gains; predicting feedback does not imply choosing good actions.
Illustrative example: An ls command already returns a directory listing. Train transition prediction from it, but deploy the policy to choose commands without first imagining a listing at each step.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
Is TAPO's training-time transition prediction the same as an extra world-model call at deployment?
Show answer and example
Answer: No. Its auxiliary objective trains a shared backbone without requiring extra prediction at deployment. Explicit world-model planning adds online calls. More training supervision need not add inference cost.
Illustrative example: The model can learn next-observation prediction and still generate only the original action before using a tool.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Why check that s_t, a_t, and the next observation belong together in TAPO targets?
Show answer and example
Answer: Misalignment or cross-episode concatenation supervises the wrong action result. Targets must come from legal feedback in the same rollout, without exposing s_{t+1} in prediction input. High mini-suite accuracy is not real tool modeling capability.
Illustrative example: Step two should predict the third observation it caused, not the first observation in the next task.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does GRSD reflect on verified success and failure trajectories for the same task?
Group-Reflective Self-Distillation
Show answer and example
Answer: Terminal reward mixes useful behavior, repeated mistakes, and chance. Contrasting reflections within a task yields specific DO/AVOID guidance to modulate turn credit.
Illustrative example: Both trajectories retrieve information, but only one checks a key condition. The contrast helps isolate useful behavior.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why generate DO/AVOID guidance with a stop-gradient snapshot and make it visible only during training?
Group-Reflective Self-Distillation
Show answer and example
Answer: Guidance comes from verified same-task successes and failures. A frozen snapshot prevents the guidance path from drifting with current gradients. It modulates turn credit without leaking hindsight group outcomes to evaluation policies. Positive multipliers preserve outcome-advantage direction rather than making reflection an answer or reward oracle.
Illustrative example: If successes check dependencies and failures edit directly, derive training guidance from that contrast. A new test task cannot receive reflections from its own future success and failure group.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does GRSD's success/failure group comparison differ from copying one successful trajectory?
Group-Reflective Self-Distillation
Show answer and example
Answer: A success can contain accidental or unnecessary steps. Group comparison uses failures to identify what to do or avoid. It is training guidance, not an oracle for independent testing.
Illustrative example: An irrelevant tool call in a successful trace is not automatically worth learning.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Why check GRSD's positive modulation m_t separately from outcome advantage?
Group-Reflective Self-Distillation
Show answer and example
Answer: m_t>0 adjusts magnitude while preserving the outcome advantage's sign; a negative value reverses the update. A stop-gradient snapshot provides training-only guidance, which must not remain in evaluation inputs.
Illustrative example: An action that should become less likely must still become less likely after multiplication by positive m.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does CAST query a solver for values of adjacent states instead of relying only on final wins or losses?
Show answer and example
Answer: Terminal signals cannot localize contributions in long games. Solver value before and after an action yields turn-level credit alongside outcome to indicate how each step changed the situation.
Illustrative example: A step can materially improve reachability before completion; its value difference supplies earlier feedback.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why does a solver's value difference between adjacent states complement rather than replace terminal outcomes?
Show answer and example
Answer: Value change provides local progress credit toward solvability and can identify important steps. Solver estimates can be imperfect and local improvement can end in failure, so retain outcome advantage as a global constraint. Any hidden answers available to the training-side solver must be isolated from evaluation actors.
Illustrative example: Simplifying constraints can earn positive local credit, but a later invalid result still makes the trajectory unsuccessful.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
Do solver state values and a learned policy value critic have the same source of supervision?
Show answer and example
Answer: A solver computes values externally in a solvable environment, while a critic usually learns estimates from experience. Solvability and query costs constrain solvers; accuracy of a tiny shortest-path oracle does not generalize to arbitrary agent tasks.
Illustrative example: Exact solutions on small deterministic graphs do not imply a similarly cheap perfect solver exists for web tasks.
Sections in the Chinese source: 背景与主要改动, 本地复现. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
How does CAST avoid reversing states in V(s_{t+1})−V(s_t)?
Show answer and example
Answer: Bind before and after states to the same action and use a consistent value direction. Reversing them records progress as negative credit. Count actual solver calls and keep training-side values from becoming a hidden-answer interface for evaluated policies.
Illustrative example: A value change from 0.2 to 0.5 should give +0.3, not −0.3.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does SEED turn reflections on completed trajectories into reusable hindsight skills?
Show answer and example
Answer: Reflections attached only to an answer do not accumulate as stable training signal. SEED extracts skills, compares probabilities for original actions with and without them, and combines dense distillation with outcome RL to reuse experience.
Illustrative example: A failure can yield the skill 'check tool preconditions first' as training guidance for later tasks.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why distill probability differences with and without hindsight skills on original actions instead of copying new expert traces?
Show answer and example
Answer: Skills come from completed-trajectory reflection. Compare conditional probabilities on states and actions actually visited by the student for denser local supervision. Hindsight explains original behavior without requiring a teacher to take another route. Deployment must not receive future answers for the same task, and skill mistakes can contaminate signals, requiring verification boundaries.
Illustrative example: If the student checked a directory and reflection finds it useful, compare the probability of that same action with and without the skill rather than substituting a teacher's command.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does SEED rescoring on original actions differ from letting a teacher generate a full successful trace?
Show answer and example
Answer: Rescoring keeps the actual student states and actions, measuring skill-conditioning changes. A new expert trace may visit different states. This is on-policy rescoring, not imitation of a separate ideal execution.
Illustrative example: If the student executed A, compare A in both conditions instead of replacing it with unexecuted B.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
What alignment must remain fixed when calculating a SEED skill log-probability difference?
Show answer and example
Answer: Use the same s_t, a_t, and policy evaluation convention, changing only skill conditioning z. Otherwise the gap includes state or action changes. Hindsight is training feedback; do not leak test-answer-derived skills into the policy.
Illustrative example: Scoring A with the skill and B without it does not measure the effect of skill guidance on A.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does TurnOPD study both rollout depth and within-turn supervision normalization?
Show answer and example
Answer: Later turns differ in training value and cost. Probes choose sampling depth, while turn-normalized KL changes supervision shares for long and short turns, separately controlling sampling budget and update weight.
Illustrative example: Sampling fewer turns and reducing the relative weight of one long turn are distinct savings.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why does normalizing KL by turn length change the training weight of long and short turns?
Show answer and example
Answer: Token sums naturally give long turns more loss, potentially drowning out short critical decisions. Turn-normalized supervision averages within a turn; separate probes allocate rollout depth to informative interactions under limited teacher budget. Weighting and sampling changes do not imply that deeper or greater supervision always helps.
Illustrative example: A key tool choice may be 10 tokens and an explanation 200. Turn averaging reduces the twentyfold weighting difference caused by length alone.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How do token-averaged KL and a sum of length-normalized turn KL differ in weighting?
Show answer and example
Answer: Token-level aggregation gives long turns more terms. Dividing each turn by its own length gives turns more comparable weights. The same token batch can still define different objectives.
Illustrative example: A 100-token turn does not automatically get ten times the weight of a 10-token turn if both are first normalized separately.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
After dynamic truncation, may TurnOPD retain teacher scores from omitted turns in the loss?
Show answer and example
Answer: Supervision must match actual cutoff L* and not use fictitious targets for uncollected turns. Respect valid response lengths. The current teacher is a deterministic task oracle, not a task-specific large model.
Illustrative example: If this rollout stops at turn three, prefabricated answers for turns four through six are not training evidence for it.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does StepPO use observation–action as the natural decision unit?
Show answer and example
Answer: Tool actions span tokens while environment feedback follows the complete action. StepPO computes value and GAE at step boundaries and aggregates action-token ratios into a step ratio before clipping.
Illustrative example: A tool call has a name and arguments, but the environment returns feedback only after the complete invocation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why does StepPO aggregate token ratios within a step before clipping the step ratio?
Show answer and example
Answer: The environment executes a whole action, so independent token clipping differs from action-level probability change. Average within-step log ratios and exponentiate, then compute value and GAE at step boundaries, aligning credit with environment transitions. Exclude tool observations from action tokens and preserve consistent length normalization.
Illustrative example: A shell command has many tokens but executes once when complete. Evaluate the step's overall change rather than assigning its state reward independently to each subword.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
Is clipping token ratios before averaging equivalent to aggregating and clipping a step ratio?
Show answer and example
Answer: No. Clipping is nonlinear, so order changes which actions are constrained. Average log ratios and exponentiate to obtain a geometric step ratio, not the arithmetic mean of raw ratios.
Illustrative example: For one large and one small ratio in an action, geometric aggregation can produce a different update from individual clipping.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
How does StepPO keep observation tokens out of action-ratio aggregation?
Show answer and example
Answer: Mark step and response-token boundaries and aggregate only policy ratios of tokens in the action. Observations are conditions, not generated actions. Deterministic local tool plans verify a step state machine but not real LLM token rollouts.
Illustrative example: A thousand-character tool response must not become a thousand policy tokens in the action's ratio mean.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does SEARL store tool experience in a graph with successful transitions?
Show answer and example
Answer: The graph represents relationships between tools and effective transitions for retrieval and policy decisions. New rollouts update edge weights and the policy, so experience changes with actual outcomes rather than remaining static.
Illustrative example: Repeated success for one tool sequence can make that relationship easier to retrieve for later tasks.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why update both graph memory and policy with new rollouts rather than freeze the tool graph?
Show answer and example
Answer: Executions reveal usable and failed tool transitions, which should change edge weights. The policy also learns to use updated relationships. Fixed graphs limit exploration to older experience, while updating the graph alone does not ensure consumption. Both sides need real rewards, not preset successful edges.
Illustrative example: If a tool's output can newly feed another tool, adding the edge is not enough; the policy should learn to select that route.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does SEARL's joint graph and policy update differ from editing memory only?
Show answer and example
Answer: Graph changes alter retrieved external information. Joint updates also change how the policy uses it and acts. Training the policy with a fixed graph likewise misses experience structure evolution. Record both state histories separately.
Illustrative example: A new successful edge does not show that policy parameters learned to use it in relevant states.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
How can SEARL show later decisions actually consume graph updates rather than only logging writes?
Show answer and example
Answer: Compare edge weights before and after rollouts, later retrieval hits, and policy inputs to confirm that updated experience is available. Nonzero telemetry proves events, not real browsing or multi-GPU RL capability from the mini-suite.
Illustrative example: If edge weights change on disk but the policy reads stale cache, the experience loop did not close.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does Simula build a factor taxonomy before generating examples?
Reasoning-Driven Synthetic Data Generation and Evaluation
Show answer and example
Answer: Direct seed expansion can limit coverage and produce similar samples. A tree organizes factors and combinations; generation controls global node combinations and local scenarios before independent critique and relative complexity calibration.
Illustrative example: Separate task type, conditions, and difficulty factors before combining them into distinct cases instead of repeatedly rewording one item.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why aren't correctness and incorrectness critiques just repeated versions of the same decision?
Reasoning-Driven Synthetic Data Generation and Evaluation
Show answer and example
Answer: A single critic may accept surface plausibility. Separately asking for evidence of correctness and looking for errors gives complementary filters. Admission requires satisfying requirements, positive correctness, and negative incorrectness judgments. Factor trees ensure coverage and batch-relative difficulty calibrates complexity; rule checks cannot replace semantic critiques as quality evidence.
Illustrative example: A polished question may contain mutually exclusive requirements. An error-seeking critic should uncover them rather than echoing 'looks good.'
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does Simula's Elo-based relative complexity differ from absolute model-assigned difficulty?
Reasoning-Driven Synthetic Data Generation and Evaluation
Show answer and example
Answer: Elo updates within-batch rankings through pairwise outcomes and depends on matchups and settings. Absolute scores use a different scale. Local initial 1000, scale 400, and k=32 are explicit settings, not claimed as the authors' unique values.
Illustrative example: Harder than other examples in this batch does not give a universal unit of difficulty across tasks.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
If Simula's two critics share a base model, may the second see the first verdict?
Reasoning-Driven Synthetic Data Generation and Evaluation
Show answer and example
Answer: No. Call independently with only the sample. Admission requires the requirements critic to pass, correct to affirm, and incorrect to deny. Semantic generation and critique are central; fixture callbacks or zero 13-gram overlap on short text do not establish full quality or absence of contamination.
Illustrative example: Telling the error critic that the other critic already passed the sample breaks independence.
Sections in the Chinese source: 核心公式, 数据协议与调用边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does DMAST share VLM parameters across agent and attacker roles?
Show answer and example
Answer: Role conditioning supports interaction and self-play with shared parameters, so robustness training covers defense decisions and attack trajectories. The full method also uses teacher imitation and oracle-reasoning SFT; the Stage 3 reward formula is one part.
Illustrative example: One base model generates both kinds of trajectories under different role prompts, not two independent policies.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why calculate group mean and variance from raw rewards while applying successful-response length penalty only in the advantage numerator?
Show answer and example
Answer: Length penalty modulates credit for one successful response, not the group's baseline and scale. Attacked groups also combine raw attacker and defender returns. Penalizing before whitening changes their comparison and the algorithm. Exclude zero-variance groups rather than using arbitrary epsilon to manufacture signal.
Illustrative example: Two equally successful responses differ in length. Penalize the longer advantage without letting extra text drag the common raw-return baseline.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
What is missing if only DMAST's group-statistics objective is implemented?
Show answer and example
Answer: It can verify loss and gradients. Full visual-agent adversarial training needs a real VLM, environment interaction, teacher imitation, oracle SFT, and self-play. A mathematical diagnostic generates no attack payload and does not establish browser success or robustness.
Illustrative example: One SGD step on a tiny policy cannot support the same capability claim as all training stages in an interactive environment.
Sections in the Chinese source: 背景与主要改动, 本地复现. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Should DMAST's training-to-sampling probability ratio multiply both PG and KL?
Show answer and example
Answer: No. Only PG receives min(π_old/π_sampling,1), and PG is zero if the ratio is below 0.1. KL is not scaled by that condition. Exclude zero-variance groups and normalize by all valid response tokens in the batch rather than averaging per role.
Illustrative example: Dropping PG for a low-ratio token does not justify deleting its reference KL.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does PEARL run offline tool trial and error before online planning?
Show answer and example
Answer: Hallucinated tools and wrong arguments break multi-hop plans. Offline exploration collects tool usage and failure conditions, then online separates Planner and Executor so planning can use more reliable experience.
Illustrative example: Learn required parameters and failure conditions before the Planner decides a call sequence.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why jointly reward tool chains and outcomes even with separate Planner and Executor roles?
Show answer and example
Answer: A well-formed plan does not ensure tools exist, order is executable, or the answer is correct. An executor can supply bad arguments despite a sound plan. Offline trial and error records usage and failures; dense online signals separately constrain tool choice, order, and result. A high process component is not actual success.
Illustrative example: Looking up an order before refunding is a correct sequence, but a wrong order ID still fails. Reward both chain quality and final outcome.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does training feedback differ between decoupled Planner/Executor and one model thinking and calling tools as it goes?
Show answer and example
Answer: Decoupling can use tool choice, order, and final success as planning rewards while the Executor handles execution feedback. A good plan does not guarantee success: verify arguments and runtime effects.
Illustrative example: A plan can pick all three tools correctly yet fail because the Executor supplies wrong arguments.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Why retain PEARL tool/order/success components rather than only total reward?
Show answer and example
Answer: Each component measures tool identity, ordering, or completion, helping localize planning signals and check weight directions. A total hides defective components. Local exploration and reuse are not live ToolHop API calls or complete GRPO.
Illustrative example: If tools are correct but ordered incorrectly, the order term should explain the lower reward.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does Agent0 create a curriculum with a task generator and several execution agents?
Show answer and example
Answer: Verifiable tasks create training opportunities automatically, multiple candidates give group feedback, and difficulty rises with the capability frontier. Zero human data does not mean no success criteria; the verifier still constrains curriculum quality.
Illustrative example: After executors master a task family, propose a slightly harder but still verifiable task.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why must task verifiers check majority votes rather than treating agreement as truth?
Show answer and example
Answer: Executors may share wrong assumptions. Voting measures agreement, not independent factual correctness. External task rules verify candidates, and curricula advance near the current ability boundary. If the generator or voter sees references, leakage invalidates claims of autonomous tool capability.
Illustrative example: Five executors can agree on the same wrong unit conversion. A verifier uses the task contract, not the number of votes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
Do agent majority agreement and independent verifier acceptance provide the same evidence?
Show answer and example
Answer: Agreement checks whether candidates match; verification checks task satisfaction. Shared errors can win a vote. Candidate groups cannot prove themselves correct, and agreement alone cannot establish improvement for curriculum advancement.
Illustrative example: Five identical wrong results still agree, but the verifier should mark failure.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Can Agent0's verifier return its internal solution or gold answer directly to executors?
Show answer and example
Answer: Execution and evaluation inputs should contain public task conditions only; hidden references stay with the verifier. Check task executability and feedback provenance. Saturated mini-suite success cannot establish LLM self-evolution.
Illustrative example: If the generator hides a canonical action sequence in the execution prompt, success no longer establishes autonomous solving.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does Agent-R1 record each agent–environment interaction separately?
Show answer and example
Answer: A transition links state, action, and next observation for pluggable context management and environments, plus selectable token- or step-level credit. It is more structured than one text example containing the whole chat.
Illustrative example: A tool call and its return form a distinct training record.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why do independent interaction transitions help switch between token-level and step-level credit?
Show answer and example
Answer: They explicitly preserve state, action, and next-observation boundaries, enabling attribution at generated-token or environment-step granularity instead of flattening the transcript. Pluggable context management cannot change transition semantics. Tool output stays an environmental observation, not a policy action loss term merely because it is logged together.
Illustrative example: A query can return a long table and still be one action. Assign step credit or split credit only among tokens that generated the query.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
Are token-level and step-level credit just two loss names?
Show answer and example
Answer: No. Decision units, value boundaries, and action-length treatment differ. Explicit transitions make choices auditable. Renaming the optimizer does not fix confused observation and policy-response boundaries.
Illustrative example: A tool action with 20 tokens is one environment step, not 20 pieces of environment feedback.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
How do Agent-R1 transition logs prevent the next observation from becoming visible to the current action?
Show answer and example
Answer: Keep temporal boundaries among s_t, a_t, and o_{t+1}; generating a_t cannot read o_{t+1}. Cut advantage recurrence at episode termination. Local mechanism updates are not actual multi-GPU agent training.
Illustrative example: Before a tool call, policy input cannot already contain its future response.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does MUA-RL include the user in dynamic rollouts instead of using only an initial request?
Show answer and example
Answer: Real users can clarify or change intent, requiring the policy to adjust tool actions. The method interacts with simulated users and learns from final completion rather than treating user input as static.
Illustrative example: A new constraint in turn two should change later tool arguments.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why exclude simulated-user messages from agent policy-token loss just like tool responses?
Show answer and example
Answer: They are produced by the environment's user policy or tools, not chosen by the agent. Supervising them confuses actions with conditions. Dynamic users support clarification and changing requirements, but final reward should measure completion rather than imitation of simulated user text. Simulation does not cover all real behavior.
Illustrative example: If a user changes the date to Friday, the agent should reschedule, not learn to generate the user's phrase 'move it to Friday' itself.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
What differs between dynamic user simulation and tool results as the only environmental feedback?
Show answer and example
Answer: User messages can change or clarify goals; tool results mainly report post-action state. Both are environment inputs with different roles. Static tool traces cannot cover policies for evolving requests.
Illustrative example: A tool can report success and a user clarify that they wanted a different outcome; the agent still needs to understand the request.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Can the MUA-RL simulated user expose the scenario's complete hidden goal directly?
Show answer and example
Answer: Only reveal what a user could say under the interaction protocol, not private references or the entire solution. Loss covers agent responses only. A deterministic local user and mini tools are not an external LLM or production database.
Illustrative example: The user can reveal a constraint after clarification rather than attaching the full canonical tool sequence at the beginning.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does Agent Lightning first convert execution events into common transitions?
Show answer and example
Answer: Agent runtimes log differently. Common MDP transitions and credit decomposition let training interpret states, actions, and rewards independently of execution, not just store logs elsewhere.
Illustrative example: A failed test after a code edit should connect to that patch action and state, not remain an isolated log message.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why is separating training and execution more than storing logs on a different machine?
Show answer and example
Answer: A common transition contract converts framework behavior into attributable MDP data for return and advantage calculation, while execution keeps its tool orchestration. Missing state/action boundaries or mixed environment tokens prevent correct updates despite physical separation. Decoupling alone does not establish improvement for any agent.
Illustrative example: A trainer can update from standardized query actions and results. An unsegmented chat transcript is not directly an action sequence.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does execution/training decoupling differ from asynchronously collecting logs?
Show answer and example
Answer: Decoupling also maps traces into trainable transitions, assigns credit, and sends updated policies back to execution. Asynchronous logging only records and transmits. Without an execution path consuming updates, the loop is not connected.
Illustrative example: Growing logs while every subsequent run uses an unchanged policy does not let training affect execution.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Which policy does the local Agent Lightning update using real test rewards?
Show answer and example
Answer: The code sandbox uses test exit codes to credit patches, updates a bounded patch policy, and reuses it by bug family. It does not run LightningRL on LLM parameters. Check command, edit, and test events separately from rewards and memory use.
Illustrative example: Failed tests can change later patch choices without updating LLM weights with RL.
Sections in the Chinese source: 本地复现, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does WebAgent-R1 train on complete multi-turn web trajectories?
Show answer and example
Answer: State, HTML, and action history change during interaction, and success depends on multiple decisions. The method manages recent and relevant context dynamically, collects full traces, and updates from group-relative outcomes instead of optimizing isolated answers repeatedly.
Illustrative example: After clicking to a new page, choose the next action from the new state, not the initial page.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why does multi-turn web GRPO need dynamic context rather than repeated single-turn answer training?
Show answer and example
Answer: Clicks, inputs, and navigation change page state. Later actions need the current page and relevant history. Full rollouts connect terminal success to generated actions, and dynamic retention limits HTML growth. Behavior cloning and initialization also shape exploration; a group-advantage formula is not complete web interaction training.
Illustrative example: After login, a changed page requires reading the new form. Repeated answers from the first screenshot cannot teach that transition.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
What training problems do behavior-cloning warm-up and subsequent M-GRPO address?
Show answer and example
Answer: Warm-up establishes basic interaction and initialization. M-GRPO uses relative full-task success to optimize the policy. The stages have distinct supervision; a GRPO registry name cannot replace the paper's behavior and long-CoT initialization.
Illustrative example: First learn legal clicks and inputs, then improve long-horizon choices from task outcomes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
How can compressed web state avoid treating old DOM as the current actionable page?
Show answer and example
Answer: Bind each action to the then-current observation and compressed context; a fixed initial page cannot substitute for dynamic state. The mini-suite uses deterministic text observations with no real DOM effects or Qwen/Llama updates.
Illustrative example: A button ID may become invalid after navigation even if it appears in history.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does GiGPO need both macro trajectory advantage and micro step advantage?
Show answer and example
Answer: Terminal outcomes summarize the trajectory but do not identify useful steps. GiGPO compares trajectories, then actions grouped by recurrent anchor state, and adds global and local credit.
Illustrative example: Two trajectories can ultimately fail while one action from their shared intermediate state is more helpful.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why compare micro advantages among actions from the same recurrent anchor state?
Show answer and example
Answer: Different states vary in available actions and remaining difficulty, confounding direct return comparisons. A shared anchor controls local conditions better. Micro credit complements trajectory-level macro advantage; stochastic continuation means differences are not noiseless causal effects or a replacement for terminal success.
Illustrative example: Compare looking for a key and pushing a locked door from the same point, not against a trajectory already at the goal.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How do same-anchor micro groups differ from ranking all steps together?
Show answer and example
Answer: A shared state makes comparisons more comparable; mixing states confounds difficulty and consequences. Shared plan steps can be local proxies but do not establish identical underlying environment states.
Illustrative example: A high return in an easy state should not be the same baseline for a difficult one.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Why record anchor identity and the corresponding action with GiGPO micro advantage?
Show answer and example
Answer: The mean comes from the same G(s_t), so wrong keys or mixed states change the baseline. Keep macro/micro logs, actual group membership, and state-proxy boundaries, not only combined advantage.
Illustrative example: Two screens with the same title can have different task state and should not be merged by title alone.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does RAGEN monitor Echo Trap separately?
Show answer and example
Answer: Multi-turn stochastic training can enter self-reinforcing repetition, with collapsed reward variance and gradient spikes. StarPO-S uses trajectory filtering, a critic baseline, and gradient/clipping stabilization rather than interpreting low variance as learning.
Illustrative example: A policy can consistently follow a failed path without any improvement.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why doesn't collapsed reward variance necessarily mean stable task learning?
Show answer and example
Answer: The policy may repeat failures or rigid behavior. Collapsed variance with gradient spikes signals Echo Trap risk. Trajectory filtering, a critic baseline, and decoupled clipping address data, estimates, and update stability separately. Real task performance is still required; low variance alone is not successful convergence.
Illustrative example: If all rollouts repeat one ineffective command, returns are zero with no variance. This is exploration collapse, not mature capability.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does StarPO-S trajectory filtering differ from clipping ratios only inside PPO loss?
Show answer and example
Answer: Filtering controls which trajectories enter the objective. Ratio clipping limits drift for admitted samples, while the critic and gradient controls are separate. One clipping operation does not replace the complete stabilization chain.
Illustrative example: Repeated low-information trajectories can still dominate training samples even if their ratios are clipped.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Does a periodic local Echo Trap probe measure the paper's training-collapse curve?
Show answer and example
Answer: No. It makes a stability path observable. The policy selects among deterministic candidates and never updates LLM parameters. Distinguish probe triggers, actual gradients, and large-scale training curves; counts are not evidence of model collapse.
Illustrative example: Checking every few rounds does not imply that the paper's Echo Trap occurred in those rounds.
Sections in the Chinese source: 本地复现, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does DeepResearcher treat search, browsing, evidence, and answers as one trajectory?
Show answer and example
Answer: Answers depend on how evidence is found and used, and final text alone obscures credit. Answer and citation rewards plus query cost balance quality and expense over the full search.
Illustrative example: One useful browse can make the answer verifiable, beyond making its final sentence fluent.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why isn't citation count enough when adding citation rewards to answer rewards?
Show answer and example
Answer: Citations connect claims to real evidence, while counts can reward repeated or irrelevant links. Search, browsing, and answering form one trajectory. Joint objectives should enforce support and correctness while penalizing wasted queries. Check claim–source links; one browse call does not establish evidence use.
Illustrative example: Ten irrelevant links should not beat two sources that directly support the conclusion, or the policy will learn link stuffing.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
Why aren't citation quality rewards and query-count penalties the same efficiency objective?
Show answer and example
Answer: Citation reward measures evidential support; query penalty controls cost. Fewer searches do not ensure reliable evidence, and more citations do not ensure useful searches. The three weights govern trade-offs; short traces alone do not establish research quality.
Illustrative example: Answering quickly after one bad query is cheap but unsupported.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
How should DeepResearcher align citation rewards with actual retrieved evidence?
Show answer and example
Answer: Citations must point to genuinely available material supporting corresponding statements, not just count as links. N_query counts actual search calls. The mini-suite has no real web or full LLM RL, so fixture citations are not external evidence.
Illustrative example: A title never retrieved and irrelevant to the claim should not receive credit for looking like a citation.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does Search-R1 repeatedly decide whether to search rather than using one-shot RAG?
Show answer and example
Answer: Intermediate evidence can expose questions or revise a belief, requiring another query. Search is an environment; reasoning and search alternate, and final answer outcomes train the policy.
Illustrative example: A first retrieval reveals entity ambiguity, so the agent can query disambiguating conditions before answering.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why mask retrieved tokens even though they occur in the input sequence?
Show answer and example
Answer: Search returns are environment output, not sampled policy actions. Policy loss there would teach the model to generate external evidence. Only its queries, reasoning, and answers are policy tokens; retrieval conditions later decisions. Iterative adaptation is not equivalent to a single static RAG retrieval.
Illustrative example: A search result that says 'published Tuesday' is not agent behavior. Train how to query and answer from evidence, not how to generate that environment text.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does visible search context differ from training on retrieved text as policy-generated tokens?
Show answer and example
Answer: Documents can condition later actions, but their tokens come from the environment, not the policy. Mask their loss. Treating them as student outputs corrupts action probabilities for policy optimization.
Illustrative example: A thousand-character webpage does not count as a thousand-character agent action with policy gradients.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
How does Search-R1's loss mask avoid masking the query action?
Show answer and example
Answer: Separate model-generated reasoning and search instructions from environment information. The former remain policy tokens and the latter get mask zero. Use provenance boundaries, not a text match for 'search.'
Illustrative example: The query inside `<search>` is an action; the following `<information>` text is an observation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Basics
Why does LOOP use several same-task trajectories instead of an independent value network?
Show answer and example
Answer: Their returns provide a leave-one-out baseline without maintaining a critic. A PPO trust region supports reuse of expensive rollouts. Removing a value network does not remove importance ratios or drift control.
Illustrative example: Compare one trajectory against the mean return of the other traces for its task.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Core mechanism
Why exclude the current trajectory from its leave-one-out baseline?
Show answer and example
Answer: Other group returns supply a critic-free reference without using the trajectory's own reward in it. Long rollouts are expensive, making token-level ratio clipping and batch reuse useful, but repeated updates still face policy drift. Old trajectories do not become indefinitely fresh on-policy data.
Illustrative example: For rewards one, one, and zero, compare the first with the mean of the other two, then clip locally drifting tokens separately.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Method comparison
How does LOOP token-level clipping differ from discarding a trajectory as soon as one token drifts?
Show answer and example
Answer: Local clipping constrains drift while retaining other useful parts of a long trace. Whole-trajectory dropping loses more expensive interaction data. Reuse must preserve old behavior probabilities; off-policy samples are not newly on-policy.
Illustrative example: One abnormal ratio in a hundred-step trajectory need not erase all other feedback.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Agents / Interactive policy learning: converting trajectory feedback into updates
Implementation details
Can LOOP compute its leave-one-out baseline directly when K=1?
Show answer and example
Answer: No. K−1 is zero, so a singleton fails the group definition and needs explicit handling or rejection. Multi-epoch reuse must preserve the corresponding old policy probabilities and episode identity, not reinsert the sample's own return into its baseline.
Illustrative example: With one example, the mean of other examples does not exist.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Topic guide and reading order
Start with this question¶
If a long task gives only a terminal success or failure, which turn should receive credit?
Takeaway¶
Connect actual environment trajectories to parameter updates. Agent Lightning separates execution from training so different frameworks can emit learnable transitions. GiGPO adds groups of revisited states for finer credit alongside trajectory-level information. World models, skill supervision, and interactive distillation also depend on state provenance. Distinguish student execution, expert demonstrations, and environment-model simulations.
Example¶
A search task has query formulation, result reading, and submission. A wrong answer could follow an off-topic query or failure to use correct evidence. Multiple actions from the same intermediate state can support a fair comparison of outcomes. Grouping unrelated states by turn number confounds task difficulty and tool behavior with action value.
Common pitfall¶
Step credit, policy updates, and capability are separate evidence levels. Finite gradients on synthetic states or three numerical seeds do not show browsing, programming, or exploration ability. A privileged training teacher or scorer may use hidden references, not the actor. Cost-based closure leaves component evidence at its original scope.
Reading order¶
Read Agent Lightning for execution, transitions, and training, GiGPO for trajectory and step credit, and ASTEROID for distinguishing student context from teacher supervision across episodes in partially observed environments.
- Agent Lightning · Original paper
- GiGPO · Original paper
- Asymmetric On-Policy Distillation Learns In-Context Exploration · Original paper
Check your understanding¶
Two third-turn actions look identical, but only one environment retrieved the right document. Can we group them for relative advantages?
Show suggested answer
Not by turn or text alone. Define and verify observable state and conditions that affect future returns. Otherwise advantage mixes environment differences. Keep execution failures and interrupted scoring to avoid selecting only successful trajectories.