Ranking invariance, calibration, and full-set recovery are distinct promises¶
简体中文 · English
Updated 2026-10-27 · 3 topic entries / 3 representative comparisons
Prism-1 constrains scoring dependencies, the JEV request audit designs paired interventions, and the multiple-correct audit checks whether a distribution recovers the correct set. They answer different questions: what the architecture guarantees, what the experiment changes, and what the evaluation measures. A valid output or one top-1 score cannot stand in for these answers.
Mechanisms · Trade-offs · Cross-paper questions · Updates
How to read this card
The table compares representative methods. It is not a shared-benchmark leaderboard. Teaching examples are not measured results.
Mechanism comparison¶
Axes: Architectural conditions for independent option scoring · Interventions on questions, evidence, and candidate eligibility · Relative odds, absolute probabilities, and set-recovery metrics
| Paper and source | Mechanism | Benefits and trade-offs |
|---|---|---|
| Prism-1: Position- and Rival-Independent Scoring for System One Models · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 代码映射与边界 |
Each option attends only to shared state, the current question, and its own block. Blocks reset to the same starting positions. Independent yes/no log-odds feed an option softmax. | Preserves relative odds among old options under fixed evidence while allowing a new winner. Does not guarantee accuracy or resistance to shared-context injection, and does not fit cross-option semantics such as “none of the above.” |
| The Company a Question Keeps: Auditing Open Jev-Style Models for Retrieval · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 已验证结果与边界 |
Separates additions to the question, displayed evidence, and candidate eligibility into paired interventions. Measures flips, ties, conditional probabilities, and allowed-label mass separately. | Measurement needs fixed public inputs and isolated scoring labels. If evidence changes, answers need not remain invariant. The local audit interface is distinct from claims about the paid models in the paper. |
| When Multiple Choices Are Correct: Asymmetric Probability Allocation in Jev and Other Decision Models · Original source Sections in the Chinese source: 背景与主要改动, 方法与本地协议, 原文结论与复现边界 |
Checks probability gaps for correct options, rankings, exact-set recovery after thresholding, and independent binary calibration to distinguish a categorical distribution from per-option correctness probabilities. | Ranking all correct options above all wrong ones does not guarantee exact recovery at a threshold. Choose thresholds on validation. The paper's model responses are not public; local work validates only scoring and isolation contracts. |
Trade-off analysis¶
Freeze the evidence before discussing independence¶
Prism-1 assumes that shared state and the question are unchanged. An eligibility-only request audit keeps all displayed evidence fixed. If a new passage clarifies the question, score changes may reflect a reasonable information update. Compare A-versus-B conditional probabilities or odds. A drop in absolute probability caused by a larger softmax denominator is not a preference reversal.
Sources: Prism-1: Position- and Rival-Independent Scoring for System One Models · The Company a Question Keeps: Auditing Open Jev-Style Models for Retrieval
Independent scores do not automatically yield multilabel probabilities¶
Even when Prism-1 makes each score independent of other options, the option softmax distributes a single unit of mass. The multiple-correct audit asks whether the entire correct set can be recovered. Correct options still compete for mass, so stable rankings do not imply that all correct options clear one threshold or that option shares are calibrated per-option correctness probabilities.
Sources: Prism-1: Position- and Rival-Independent Scoring for System One Models · When Multiple Choices Are Correct: Asymmetric Probability Allocation in Jev and Other Decision Models
Cross-paper questions¶
Try answering before expanding the explanation. Each question relates at least two papers.
If A's probability drops when an unrelated C is added, has the property Prism-1 protects already been violated?¶
Show answer and example
Answer: Not necessarily. Fix the evidence, then compare A-to-B odds or A/(A+B). C raises the denominator, so A's absolute share can drop normally. Only a change in the old pair's relative preference calls for investigating cross-option information or position dependence. The request audit separates this normalization effect from added evidence.
Illustrative example: With exponentiated scores 2 and 1, A has 2/3 of the mass. Adding C with score 5 lowers A's share to 2/8, but A/B remains 2; the old candidates have not reversed.
Sources: Prism-1: Position- and Rival-Independent Scoring for System One Models · The Company a Question Keeps: Auditing Open Jev-Style Models for Retrieval
Why is ranking every correct option ahead of every incorrect option insufficient to solve a multiple-answer question?¶
Show answer and example
Answer: A ranking gives relative order, not how many options to select. Option softmax distributes one unit of mass, so the second correct option may have little mass. A fixed threshold may omit it, and lowering the threshold may admit an incorrect option. Measure exact-set recovery separately and select thresholds on independent validation data.
Illustrative example: A and B are correct and the shares are 0.60, 0.25, and 0.15. The ranking is perfect, but a threshold of 0.4 returns only A. Correct top-1 does not expose the missing B.
Sources: When Multiple Choices Are Correct: Asymmetric Probability Allocation in Jev and Other Decision Models · Prism-1: Position- and Rival-Independent Scoring for System One Models
Why must adding C as an eligible candidate and adding evidence describing C be tested separately?¶
Show answer and example
Answer: Eligibility changes the action set; evidence changes what the model knows. Prism-1 constrains independent option scores under fixed shared evidence only. The request audit separates interventions to distinguish competition effects from legitimate information updates. Multiple-correct scores must also be interpreted under fixed input conditions and a fixed correct set.
Illustrative example: If tool C is already described in full, making it selectable changes eligibility. Adding “C supports offline operations” introduces new evidence that can change the A/B preference.
Sources: The Company a Question Keeps: Auditing Open Jev-Style Models for Retrieval · Prism-1: Position- and Rival-Independent Scoring for System One Models
Update log¶
Updates record new approaches, changed trade-offs, and follow-up evidence.
2026-10-27 · Initial version: compared Prism-1, the JEV request audit, and the multiple-correct audit. Covered independent option scoring, interventions on questions, evidence and eligibility, and cross-paper questions.
Affected: When Multiple Choices Are Correct: Asymmetric Probability Allocation in Jev and Other Decision Models · The Company a Question Keeps: Auditing Open Jev-Style Models for Retrieval · Prism-1: Position- and Rival-Independent Scoring for System One Models
All 3 topic entries
The full catalog follows. The main comparison covers the representative papers listed above.