Evidence of capability: scoring accurately, observing less, and explaining predictions¶
简体中文 · English
Updated 2026-11-04 · 14 topic entries / 6 representative comparisons
Distinguish how correctness is defined, which example to test next, how judges are combined, and what an explanation says. XLGoBench, ProEval, FUSE, TCAB, and DetectorScope change different parts. Fewer observations and additive explanations do not imply stronger underlying capability. XLA from the Trenches adds a prior concern: compiled execution can silently depart from numerical program semantics. Successful compilation does not establish correct execution.
Mechanisms · Trade-offs · Cross-paper questions · Updates
How to read this card
The table compares representative methods. It is not a shared-benchmark leaderboard. Teaching examples are not measured results.
Mechanism comparison¶
Axes: Correctness labels, posterior overall scores, rankings, or additive explanations · When labels become available and who holds them · Deterministic algorithms, cross-model correlations, or verifier identification assumptions · Saving scoring calls, duplicate feedback, or explanation cost
| Paper and source | Mechanism | Benefits and trade-offs |
|---|---|---|
| XLGoBench: Detecting cross-lingual skill gaps with algorithmic tasks · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Use deterministic algorithm tasks and exact scorers; the model sees the task/problem, never scorer gold. | Measures specified families and complexities, not open QA. The five-task local slice omits the full language and complexity matrix. |
| ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 代码映射与边界 |
Build an empirical GP from historical model-by-question correlations and choose informative observations for target-score estimation. | Relies on transfer of historical relations. Target scores appear only after queries. A four-question math fixture does not establish real evaluation savings. |
| FUSE: Ensembling Verifiers with Zero Labeled Data · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 本地复现与成本判断 |
Estimate verifier quality and correctness posteriors from a score matrix, then learn an ensemble; external evaluation keeps true labels. | Assumptions such as better-than-random majority and conditional independence can fail with correlated judges. No local ensemble implementation. |
| Fast A/B/n Testing: Exact Multi-Policy Comparison via Tree-Coupled Feedback Sharing · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Use maximal coupling to share feedback when adaptive policies make identical choices, organizing shared decisions as a tree while preserving marginal trajectories. | Correct coupling is required; outcomes cannot be copied to different actions. Local mini-suite does not establish asymptotic savings or production A/B gains. |
| Building Auditable AI Text Detectors with Sparse Autoencoders · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 代码映射与边界 |
Screen external SAE concepts and apply a sparse linear head with logit contributions verifiable by zeroing pooled features. | Feature attribution is not a causal effect of rewriting text. Local synthetic activations validate the L1 component, not a full detector or SAE. |
| XLA from the Trenches: Correctness on TPU Workloads at Scale · Original source Sections in the Chinese source: 核心机制与数值例子, 论文工程与线上证据, 资源与复现边界 |
Check differences on valid inputs under target-TPU numerics, isolate large graphs, then inspect VREGs inside intact fusions without debugging away the fault. | Compiler correctness is not a model-capability score. Private production graphs and TPU tools are unavailable; public OpenXLA covers only part of the pipeline. Local status is review-only, with no CUDA surrogate. |
Trade-off analysis¶
What do deterministic scoring and probabilistic surrogates answer?¶
XLGoBench computes exact algorithmic answers for structured outputs. ProEval keeps task correctness fixed and predicts overall performance from observed scores. Posterior means and variances come from model relationships, not real scoring of every question. Combine the ideas by isolating scorer gold, then deciding which legitimate scores to pay for while separating inferred and observed outcomes.
Sources: XLGoBench: Detecting cross-lingual skill gaps with algorithmic tasks · ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation
An extra layer of evidence is not automatically reliable¶
FUSE uses multiple verifiers without true labels, but correlations and identification assumptions govern its posterior. DetectorScope decomposes logits exactly but shows only numerical contributions of pooled features, not truthful concepts or causal text effects. TCAB constrains marginal distributions under feedback reuse. These checks address different issues, and neither fluent explanations nor reuse rates verify capability.
Sources: FUSE: Ensembling Verifiers with Zero Labeled Data · Building Auditable AI Text Detectors with Sparse Autoencoders · Fast A/B/n Testing: Exact Multi-Policy Comparison via Tree-Coupled Feedback Sharing
The reference must match execution semantics too¶
XLGoBench separates scorers from gold. XLA shows that even valid tasks and labels do not guarantee compiled computation preserves program semantics: register rounding or fusion scheduling can change results. Differential references must exclude NaN-producing invalid inputs without flagging legitimate TPU approximations as bugs. A correct split fusion does not validate the intact fusion, and correct A100 computation does not validate TPU lowering. This batch preserves the missing TPU environment instead of replacing hardware conclusions with CPU/CUDA demonstrations.
Sources: XLGoBench: Detecting cross-lingual skill gaps with algorithmic tasks · XLA from the Trenches: Correctness on TPU Workloads at Scale
Cross-paper questions¶
Try answering before expanding the explanation. Each question relates at least two papers.
Why must sparse ProEval testing still isolate real scorers and gold answers as XLGoBench does?¶
Show answer and example
Answer: ProEval changes measurement selection and total-score estimation; observations still need legitimate task scoring. Leaking answers to the model or unqueried scores to the posterior contaminates results. Keep predictions, revealed scores, and scorer-internal gold separate.
Illustrative example: Query item two before reading its score. Initializing with all target scores defeats a small-query evaluation.
Sources: ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation · XLGoBench: Detecting cross-lingual skill gaps with algorithmic tasks
Why are FUSE correctness posteriors and DetectorScope concept contributions not direct truth proofs?¶
Show answer and example
Answer: FUSE estimates correctness under verifier behavior assumptions and can inherit correlated errors. DetectorScope measures the logit change when a pooled feature is zeroed, not whether a concept story is true or causally supported by text. Validate statistical reliability and explanations independently.
Illustrative example: Judges that all prefer long answers can share a mistake, and a strong detector feature may reflect a dataset shortcut.
Sources: FUSE: Ensembling Verifiers with Zero Labeled Data · Building Auditable AI Text Detectors with Sparse Autoencoders
Why not record TCAB shared feedback and ProEval imputed scores as the same observations?¶
Show answer and example
Answer: TCAB reuses real feedback only on valid coupled identical decisions while preserving each policy's marginal process. ProEval predicts unqueried scores using historical covariance. Neither licenses reporting a different action's outcome or an untested score as executed.
Illustrative example: Two policies choosing action A can share A's feedback. If another chooses B, its correlated history does not authorize assigning A's success to B.
Sources: Fast A/B/n Testing: Exact Multi-Policy Comparison via Tree-Coupled Feedback Sharing · ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation
With exact XLGoBench gold, why doesn't completing a program guarantee trustworthy model evaluation?¶
Show answer and example
Answer: Exact gold defines correct output, not whether compiled model execution preserves semantics. XLA demonstrates silent numerical changes without crashes. Valid inputs, hardware-matched differential references and isolation are still needed. Check label correctness separately from execution correctness to distinguish model errors from backend errors.
Illustrative example: Teaching example: Gold requires a nonzero quantization residual, but a TPU fused kernel outputs zeros after skipping casts. An exact scorer can mark the output wrong; it cannot automatically conclude the model lacks the ability.
Sources: XLGoBench: Detecting cross-lingual skill gaps with algorithmic tasks · XLA from the Trenches: Correctness on TPU Workloads at Scale
Update log¶
Updates record new approaches, changed trade-offs, and follow-up evidence.
2026-11-04 · Add Google's XLA:TPU compiler-correctness research, distinguishing valid scoring, reference hardware semantics, silent miscompilation and engineering bug counts. Full internal TPU experiments are too costly and not reproduced; review DONE.
Affected: XLA from the Trenches: Correctness on TPU Workloads at Scale
2026-10-27 · Initial version: compared exact algorithm scoring, probabilistic evaluation surrogates, unlabeled verifier ensembles, feedback coupling, and additive concept attribution, separating observation from inference.
Covers 13 topic entries; see the full list below.
All 14 topic entries
The full catalog follows. The main comparison covers the representative papers listed above.
- XLA from the Trenches: Correctness on TPU Workloads at Scale
- Detecting Safety Training Modification in Language Models via Activation Analysis
- Detecting Post-generation Edits to Watermarked LLM Outputs via Combinatorial Watermarking
- CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes
- Cross-Lingual Exploration for Parametric Knowledge
- DeepSEA: A Multilingual Evaluation Benchmark for Southeast Asia
- Building Auditable AI Text Detectors with Sparse Autoencoders
- FUSE: Ensembling Verifiers with Zero Labeled Data
- Words Speak Louder Than Order: A Behavioral Evaluation of Gemma 4
- How Calibration Content Shapes Attention-Based Reranking
- Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders
- ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation
- Fast A/B/n Testing: Exact Multi-Policy Comparison via Tree-Coupled Feedback Sharing
- XLGoBench: Detecting cross-lingual skill gaps with algorithmic tasks