Four evaluation questions: Was it completed, is it robust, why did it work, and what did it cost?¶

Updated 2026-11-04 · 44 topic entries / 7 representative comparisons

GAIA scores final answers. AgentPerturb tests single-target input changes; User Proxy Bench checks whether simulators bypass information gathering. AlphaDiana tracks failures and retries; HDA decomposes harness gains, and selective evaluation bounds automated judge error. High scores only become interpretable after these protocols are explicit.

Mechanisms · Trade-offs · Cross-paper questions · Updates

How to read this card

The table compares representative methods. It is not a shared-benchmark leaderboard. Teaching examples are not measured results.

Mechanism comparison¶

Axes: Whether the target is final output, input robustness, user contracts, or automated-judge risk · Whether paired interventions change input, compute, or detected shortcuts · Whether ungraded cases, retries, and shared memory remain in denominators and costs · Whether confidence intervals and certificates use independent tasks or correlated trajectories

Paper and source	Mechanism	Benefits and trade-offs
GAIA · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Scores real-tool and multimodal problems by normalized short-answer exact match with three difficulty levels.	Final correctness does not explain process or assistance. Local gaia-mini tests the protocol, not the original 466-task capability.
AgentPerturb: Evaluating LLM Agent Robustness under Controlled Interaction Perturbations · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 真实性、gold 隔离与成本	Crosses four input surfaces with three perturbation types, changes one target with independent solvability verification, and compares paired tasks across both sides.	Syntax does not establish fidelity. Report coverage and uncertainty bounds for ungraded cases. Local arithmetic fixtures are not the official environment matrix.
UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training · Original source Sections in the Chinese source: 背景与主要改动, 本地复现	Defines UFS as the product of applicable user criteria and separately audits private-field disclosure before requests.	Task reward is not proxy fidelity. Early disclosure can improve success without testing active inquiry. Local work checks rubrics and event order only.
AlphaDiana: A Measurement System for Agents in Verifiable Reasoning Tasks · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 无 gold 边界与限制	Tracks attempts under stable logical samples, retains ungraded status, sums retry costs, and compares model–harness systems under a common policy.	Confirmed-correct share is not the unknown-completed true success rate. Missing costs preclude complete tokens per success. Local work is synthetic accounting.
Harness-Delta Attribution: Dissecting Gains From Harness Evolution · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 本地复现与边界	Tests paired gain first, then constructs four arms with human-reviewed compute matching and shortcut neutralization to separate compute, detected shortcuts, and residual.	Neutralization only covers discovered mechanisms; the residual may still include shortcuts and is not direct transfer evidence. Local work runs statistics on measured arms only.
Certified Selective Automation of LLM Agent Evaluation · Original source Sections in the Chinese source: 背景与主要改动, 复现边界	Bootstraps by task cluster to estimate an upper bound on error in an automated region and automates only when certification meets budget.	Depends on calibration clusters and the targeted automation region; trajectories are not independent samples. Mini certificates do not establish corpus coverage or general judge reliability.
AgentHeal: Evaluating Goal-Preserving Resource Recovery of Web Agents under Access Barriers · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 数据协议与实现边界, 本地结果与负控	Constructs barrier-free, recoverable, and unrecoverable resource triples, jointly evaluating goal preservation through copy/version relations, barrier diagnoses, and access policy.	Honest abstention, not forced answering, is appropriate on unrecoverable tasks. Independent provenance and policy annotations are required; local truth tables provide neither browser nor semantic-judge capability.

Trade-off analysis¶

Task success can conceal a change in what is being measured¶

GAIA-style exact answers show whether deliverables match references. If a simulated user reveals information the agent was supposed to ask for, the task can succeed while testing less information gathering. User Proxy Bench separates UFS from reward. AgentPerturb likewise requires unchanged intent and solvability: blaming agents for broken tasks does not measure robustness. Good evaluation controls the task and visibility, not just whether scores move.

Sources: GAIA · UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training · AgentPerturb: Evaluating LLM Agent Robustness under Controlled Interaction Perturbations

Paired designs still need failure records and attribution conditions¶

AlphaDiana preserves logical sample counts across retries, sums attempt costs, and separates successful execution from grading. HDA aligns four arms itemwise, controls budgets, uses human-reviewed neutralization, and establishes gains before attribution. Selective evaluation clusters repeated trajectories by task instead of treating them as independent calibration. These govern interpretation without converting missing grades, unseen shortcuts, or uncovered tasks into verified outcomes.

Sources: AlphaDiana: A Measurement System for Agents in Verifiable Reasoning Tasks · Harness-Delta Attribution: Dissecting Gains From Harness Evolution · Certified Selective Automation of LLM Agent Evaluation

When an unsolvable task is valid evaluation rather than artificially broken robustness¶

AgentPerturb preserves the original objective and solvability after perturbation, avoiding manufactured failures. AgentHeal instead labels genuine unrecoverability as its own regime, assessing honest abstention, diagnosis, and compliance without demanding nonexistent answers. Recoverable cases still require preferring exact copies or disclosing version differences. GAIA-style final-answer correctness does not replace source and process conditions. Answer rates, recovery rates, and reliability stay distinct only when expected behavior and denominators are explicit for each regime.

Sources: AgentHeal: Evaluating Goal-Preserving Resource Recovery of Web Agents under Access Barriers · AgentPerturb: Evaluating LLM Agent Robustness under Controlled Interaction Perturbations · GAIA

Cross-paper questions¶

Try answering before expanding the explanation. Each question relates at least two papers.

Why can User Proxy Bench reject evaluation even when a tool task succeeds, and why require solvability after AgentPerturb changes?¶

Show answer and example

Answer: User Proxy Bench checks disclosure contracts: giving private fields unprompted bypasses clarification ability. AgentPerturb preserves goal, executability, and solvability so impossible tasks do not masquerade as robustness losses. Final accuracy measures delivery; these protocols separately constrain excess assistance and excess disruption.

Illustrative example: Providing an order number up front skips asking for it. Removing the number and forbidding clarification is also an unfair task.

Sources: GAIA · UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training · AgentPerturb: Evaluating LLM Agent Robustness under Controlled Interaction Perturbations

If the perturbed grader fails, the clean arm is correct, and there are two retries, how do AgentPerturb and AlphaDiana avoid flattering scores?¶

Show answer and example

Answer: AgentPerturb computes paired differences on matching doubly scored items and still reports coverage and unknown bounds over all requests. AlphaDiana keeps retries under one logical sample, chooses latest rather than best, and sums all known attempt costs. Missing grades or tokens must not be fabricated into full success cost.

Illustrative example: Three costly attempts ending with grader failure cannot be reduced to one prior success or removed from the planned denominator.

Sources: AgentPerturb: Evaluating LLM Agent Robustness under Controlled Interaction Perturbations · AlphaDiana: A Measurement System for Agents in Verifiable Reasoning Tasks

Why do remaining HDA gains after neutralizing shortcuts and passing a selective-evaluation risk certificate support different conclusions?¶

Show answer and example

Answer: HDA compares four arms and attributes residual gain after compute and detected shortcuts, which can still include unknown shortcuts and does not prove transfer. Selective evaluation bounds error in an automated region using calibration clusters; it does not explain why a harness improved. Both require task scope and statistical units.

Illustrative example: A judge meeting its regional error bound does not show the agent improved through reasoning. Remaining gains after removing one shortcut do not ensure judges transfer to another domain.

Sources: Harness-Delta Attribution: Dissecting Gains From Harness Evolution · Certified Selective Automation of LLM Agent Evaluation

Why can AgentHeal reward abstention on unrecoverable tasks while AgentPerturb cannot arbitrarily break tasks until they are unsolvable?¶

Show answer and example

Answer: Their predefined objectives differ. AgentPerturb measures paired robustness under the same solvable objective. AgentHeal measures resource recovery, explicitly distinguishing recoverable and unrecoverable cases; the latter require diagnosis and honest stopping or user assistance. Such abstention is not final-answer accuracy, and an aggregate score must not conceal different regimes.

Illustrative example: Teaching example: when the only file disappears without a lawful backup, describing the barrier instead of fabricating content is a reliability success. Requiring the original number on that same condition as a perturbation task instead breaks the question.

Sources: AgentHeal: Evaluating Goal-Preserving Resource Recovery of Web Agents under Access Barriers · AgentPerturb: Evaluating LLM Agent Robustness under Controlled Interaction Perturbations · GAIA

Update log¶

Updates record new approaches, changed trade-offs, and follow-up evidence.

2026-11-04 · Added AgentHeal against AgentPerturb and GAIA to explain unrecoverable regimes, requested versions, attribution, and the policy product. Distinguished the exact-copy intersection from the other-version subset condition, with gold annotations visible only to evaluators. Local TS=0.5 is a four-case synthetic truth table, not agent accuracy. The 855-task, 13-model full matrix is not reproduced because of high cost, review DONE.

Affected: AgentHeal: Evaluating Goal-Preserving Resource Recovery of Web Agents under Access Barriers · AgentPerturb: Evaluating LLM Agent Robustness under Controlled Interaction Perturbations · GAIA

2026-10-27 · Initial version: compared endpoint capability, single-target perturbations, user contracts, failure/retry accounting, four-arm attribution, and cluster-level selective certification.

Covers 43 topic entries; see the full list below.

All 44 topic entries

The full catalog follows. The main comparison covers the representative papers listed above.

Paper-by-paper questions · All deep dives