From making calls to delivering results: Tool interfaces, reliability, and collaboration contracts¶

Updated 2026-10-29 · 49 topic entries / 6 representative comparisons

Toolformer learns when to call, SWE-agent designs action interfaces, and RATED addresses reliability among relevant tools. AutoGen programs messaging, MetaGPT structures role deliverables, and World of SWEcraft checks collective output on hidden downstream dependencies. Any layer can fail; a valid call format is not end-to-end completion.

Mechanisms · Trade-offs · Cross-paper questions · Updates

How to read this card

The table compares representative methods. It is not a shared-benchmark leaderboard. Teaching examples are not measured results.

Mechanism comparison¶

Axes: Whether improvement targets call policy, interaction interface, tool selection, or team protocol · Whether feedback comes from language loss, actual execution, structured deliverables, or hidden downstream tasks · Whether collaboration state is dialogue, explicit intermediate artifacts, or persistent repository assets · How failures and resources are included in delivery metrics

Paper and source	Mechanism	Benefits and trade-offs
Toolformer · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Samples API candidates, filters them by how actual returns improve later token loss, and continues language-model training.	Training utility does not imply reliable deployment tools. A deterministic local utility stands in for GPT-J loss, without full continued training.
SWE-agent · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Uses a dedicated ACI for search, viewing, precise editing, and testing, making software tasks repeatable action loops.	A lower interaction burden does not replace comprehension or verification. Real local temporary repositories form a microbenchmark, not official SWE-bench resolved rates.
AutoGen · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Programs messages with ConversableAgent, reply policies, and stop conditions, supporting group chats and nested conversations.	Flexibility still needs protocol design. More roles do not guarantee better collaboration. The local path executes messaging and handoff, not the full runtime.
MetaGPT · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Encodes software SOPs into role workflows, with structured intermediates at each step for downstream use and verification.	Deliverable structure limits free-form chat but downstream roles can inherit upstream errors. Real local editing and tests are not a full LLM team.
RATED: Reliability-Aware AI Agent Tool Retrieval via Execution Dynamics · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 结果、验证与边界	Uses frozen dual encoders for semantic Top-K, then reranks using metadata and per-tool discounted LinUCB from actual feedback.	Only chosen tools have observed feedback; uncalled tools cannot receive fabricated returns. Short encoder training ran locally, but authorization binding failed and original ecosystem capability was not established.
World of SWECraft: Agent Collaboration at Scale via Simulated Software Worlds · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 本地复现与成本判断	Creates a software world of persistent sessions, issues, PRs, and package dependencies. Hidden downstream projects install frozen versions to assess collective artifacts.	Local microbenchmark success does not establish downstream gains, and instruction counts are not wall time. The expensive full 55K-session environment did not run locally.

Trade-off analysis¶

Choosing a tool does not establish that it works¶

Toolformer filters calls by prediction gains, supplying policy-training signal. SWE-agent organizes executable interfaces and their return formats. RATED separates semantic relevance from recent reliability. Record willingness to call, valid format, actual interface success, and task completion independently. Relevance does not compensate for a broken service, and successful calls do not supply counterfactual rewards for unchosen tools.

Sources: Toolformer · SWE-agent · RATED: Reliability-Aware AI Agent Tool Retrieval via Execution Dynamics

Messaging protocols, intermediate artifacts, and ecosystem effects¶

AutoGen programs who replies and when to continue or stop. MetaGPT further constrains role inputs and outputs with SOP deliverables. World of SWEcraft widens evaluation to long-term maintenance across repositories: package tests can pass while downstream consumers break. These methods address control, information handoff, and external effects respectively. Counts of messages, roles, or artifacts do not replace downstream evidence. Long-lived collaboration needs persistent assets and session budgets rather than an indefinitely growing chat.

Sources: AutoGen · MetaGPT · World of SWECraft: Agent Collaboration at Scale via Simulated Software Worlds

October 29: New method and source changes¶

Meta's official retrieval-programs repository is now verifiable, updating RPL's prior open-source record. The main repository is CC-BY-NC4; vendored retrieval and training components retain their licenses. Public code does not make the local typed grammar cover arbitrary Python, cross-turn state, neural retrieval, or RL. No source was copied and the existing L1 evidence did not change.

Sources: Toolformer · Learning to Program Retrievals: Composing Search Primitives via Reinforcement Learning

Cross-paper questions¶

Try answering before expanding the explanation. Each question relates at least two papers.

Why might RATED deprioritize a tool whose API call Toolformer selected as useful?¶

Show answer and example

Answer: Training selection shows that returned values helped prediction, not continued service availability or stable argument behavior. RATED first restricts to relevant tools, then updates reliability from selected tools' actual successes, errors, and timeouts using discounting. Training usefulness and current executability are separate, and one failure cannot assign negative rewards to all untried tools.

Illustrative example: An exchange-rate API can help training and time out repeatedly in deployment. It remains relevant, but actual reliability should affect future choices.

Sources: Toolformer · RATED: Reliability-Aware AI Agent Tool Retrieval via Execution Dynamics

If AutoGen already has programmer and tester roles, why do MetaGPT and SWE-agent emphasize structured deliverables and ACI?¶

Show answer and example

Answer: Role names specify responsibilities, while AutoGen's replies and stopping organize interaction. MetaGPT specifies deliverables for requirements, design, and implementation to reduce drift in free chat. SWE-agent defines concrete viewing, editing, and testing actions. Control, information contracts, and action interfaces are complementary, not interchangeable.

Illustrative example: A tester can know when to reply without a clear interface specification. Even with a complete spec, an imprecise editing command can change the wrong code.

Sources: AutoGen · MetaGPT · SWE-agent

Why does World of SWEcraft use hidden downstream packages after MetaGPT or SWE-agent repository tests pass?¶

Show answer and example

Answer: Own tests cover maintainer-visible contracts and may only improve visible microbenchmarks. API changes can break real consumers. Hidden downstream checks install frozen release versions to assess collective delivery. Retain failed packages in speed accounting and declare strict/carry semantics. A local micro pass does not establish ecosystem capability.

Illustrative example: Renaming an argument and updating library tests can still break consumers using the old name; downstream checks reveal it.

Sources: MetaGPT · SWE-agent · World of SWECraft: Agent Collaboration at Scale via Simulated Software Worlds

Why retain local RPL component limitations after finding the official repository?¶

Show answer and example

Answer: Source completeness and execution fidelity are different. Published training code does not make the existing constrained SDK execute the full policy, neural retrieval, or distillation. Future integration needs separate cost, execution, and license checks.

Illustrative example: Learning the official training entry point does not turn local retrieval-program syntax tests into multi-hop QA evaluation.

Sources: Toolformer · Learning to Program Retrievals: Composing Search Primitives via Reinforcement Learning

Update log¶

Updates record new approaches, changed trade-offs, and follow-up evidence.

2026-10-29 · Meta's official retrieval-programs repository is now verifiable, updating RPL's prior open-source record. The main repository is CC-BY-NC4; vendored retrieval and training components retain their licenses. Public code does not make the local typed grammar cover arbitrary Python, cross-turn state, neural retrieval, or RL. No source was copied and the existing L1 evidence did not change.

Affected: Learning to Program Retrievals: Composing Search Primitives via Reinforcement Learning

2026-10-27 · Initial version: compared call learning, ACI, message orchestration, SOP deliverables, reliability reranking, and hidden downstream evaluation.

Covers 49 topic entries; see the full list below.

All 49 topic entries

The full catalog follows. The main comparison covers the representative papers listed above.

Paper-by-paper questions · All deep dives