ToolRL: Multi-Tool Learning with Execution Rewards

简体中文

Core mechanism evidence. Paper and local evaluations are separate.

Paper information

Field	Value
Paper	ToolRL (arXiv 2504.13958)
Affiliations	University of Illinois Urbana-Champaign
First public release	2025-04-16 (arXiv v1)
Authors' code	No official author repository found or released.
Adapter / method	toolrl
Local code	src/auto_research/agent_research/

Original paper

Background and contribution

Jointly optimize tool selection, arguments, and execution results. Dynamic reward scaling mixes tools of different difficulty within an RL batch.

flowchart LR
  T[Tool choice] --> A[Arguments] --> E[Execution] --> R[Tool-normalized rewards]

Key figure from the paper

Original Figure 1: ToolRL. From the paper; copyright belongs to the authors. Not a local result.

Core mechanism

\[ R=R_{select}+R_{args}+R_{exec}+R_{answer},\quad\hat A=(R-\mu_{tool})/(\sigma_{tool}+\epsilon). \]

Paper results

The paper improves tool choice, arguments, and final success on multi-tool benchmarks. No production A/B.

Local reproduction

PlanBench mini, 120 episodes, seed 42: joint success 1.0000, average cost 1.1250, nonzero method telemetry. Metrics.

auto-research agent-eval --method toolrl --benchmark planbench-mini --episodes 120 --seed 42
auto-research evolve --model agent --dataset planbench-mini --direction "Combine toolrl with installed paper operators" --generations 2 --population 4

Reproduction limits

Only the mechanism and budget protocol; no paper-scale LLM, distributed RL, private environments, real web, or full benchmark.