EvalAlloc: Fixed-reference candidate scoring and paired confirmation¶
Local scope: CPU-only statistical and budget-planning subcomponents. This is not the full EvalAlloc search controller. No LLM calls or fabricated task scores are used. The complete search and benchmark experiments were reviewed and not reproduced because of cost.
Paper information¶
| Field | Details |
|---|---|
| Paper | Where Should the Evaluation Budget Go in LLM-Driven Agent Scaffold Evolution? |
| Reviewed version | arXiv v1 full text |
| First author and affiliation | Sichun Luo, University of Hong Kong; collaborators include China Mobile JIUTIAN |
| First public release | 2026-10-27 14:06:26 UTC, arXiv v1 |
| Author code | No verifiable EvalAlloc repository found in the reviewed v1 sources. Referenced frameworks such as ShinkaEvolve are not EvalAlloc code. |
| Local key | evalalloc; statistical and paired-round component, not integrated into Evolve |
| Local implementation | src/auto_research/agent_research/evalalloc.py |
What the paper proposes¶
Search needs inexpensive, comparable feedback; selecting the final winner needs stronger confirmation from actual measurements. EvalAlloc fully evaluates a few warm-up candidates, selects shared informative items, and fits an item-response model. During search, it keeps the item model fixed and infers only a new candidate's parameters. Observed scores are then combined with predictions for unobserved cells. Final confirmation expands a common random item prefix in rounds and reuses cached observations.
Original paper figure¶
Unredrawn figure from the reviewed paper; see the figure provenance. The pictured item-model training, full evolution loop, and multi-round successive halving are not implemented locally.
Core equations¶
For item \(i\) with \(a_i\) full-credit observations out of \(n_i\) total, \(p_i\sim\operatorname{Beta}(a_i+1,n_i-a_i+1)\). The conservative information score is
Only full credit contributes to \(a_i\). Partial credit remains a score for response modeling but counts as not full credit in acquisition. Negative information scores are not clipped. Outside the shared core, Gumbel top-\(k\) uses logits \(I_i/\tau\) for sampling without replacement.
With item parameters and the Gaussian prior frozen, MAP fits the new candidate using its sparse observations. Predictions never overwrite known scores. In confirmation round \(r\), \(U_c^r=O_c\cup P_r\) combines cached observations with the common prefix; only missing cells are evaluated. The first round uses the union mean. Later rounds use
where the pooled within-candidate regression slope is clipped to \([0,1.5]\). The paper refits MIRT in two cross-fitted folds after the first round. The local component accepts externally fitted predictions and training-candidate identities; it rejects a prediction fitted using the candidate being scored. When every item is observed, the correction vanishes exactly and the final result rests on observations.
What the paper reports¶
The authors compare evaluation budgets on four benchmarks, two evolution frameworks, and three runs, reporting competitive or higher mean scores at roughly half the cell budget. Table 5 shows that cell count and token cost do not scale proportionally, so halving cells is not the same as halving LLM spend. The local component does not run the paper's searches or report TEST scores. No production A/B test is reported.
Local reproduction and validation¶
The interface receives SEARCH item indices and sparse observations, not DEV/TEST answers or a hidden complete score matrix. FrozenItemModel copies externally fitted item parameters and the prior into read-only arrays. Only new candidate parameters change during MAP. Callers remain responsible for checking the actual fitting data and cross-fit provenance; supplied identities do not by themselves prove a leak-free training process.
informativeness implements Eqs. 2–3; shared_core and acquire cover Eq. 5; infer_candidate fits MAP with a fixed external model; completed_fitness covers Eq. 6; regression_adjusted_scores covers Eqs. 7–9. plan_paired_round and accept_round_results check the missing-cell budget and validate the returned cells before caching. L-BFGS tolerances and iteration limits are local solver settings, not unpublished author settings.
Warm-up MIRT fitting, two-fold refitting, AST deduplication, full multi-round successive halving, model calls, and evolution-framework integration are out of scope. The CPU numerical kernel requires the research-components extra for SciPy, not PyTorch, a checkpoint, or a GPU.
From the repository root:
python -m pip install -e '.[research-components]'
PYTHONPATH=src python -c 'from auto_research.agent_research.evalalloc import diagnostic; print(diagnostic(42))'
With the repository test dependencies installed, run PYTHONPATH=src python -m pytest tests/test_oct30_post_agent.py -q. The tests compare closed-form Beta moments, verify frozen item parameters and observation precedence, check shared-core selection and sampling without replacement, preserve the cache union, reject candidate contamination and missing-cell budget violations, and require zero correction under full observation. See the three-seed CPU record. It contains mechanism statistics, not agent success or actual savings.
Full experiment: reviewed, not reproduced¶
The paper's matrix covers 80 or 81 proposals, 300 SEARCH items per benchmark, four benchmarks, two frameworks, three runs, multiple LLM baselines, DEV/TEST audits, and a proposal model up to 120B. The full experiment was not launched, and omitted fitting or controller pieces are not registered as Evolve integrations.