Candidate Invariance: Order, Competitors, and Calibration¶
简体中文 · English
Structured Decisions / 12 questions · 3 papers
Try answering each question before opening the explanation. The bank grows as papers and questions are added.
Read this topic's comparisons, analysis, and cross-paper Q&A
12 questions · 3 papers
Structured Decisions / Candidate Invariance: Order, Competitors, and Calibration
Basics
Does a multicorrect task require one answer or the entire correct set?
Show answer and example
Answer: Recover all correct candidates; there may be more than one. One correct choice can miss others; choosing everything can include distractors. Interpret ranking, thresholding and exact-set metrics using a set rather than a winner.
Illustrative example: Example: Correct {A,B} is not recovered by {A} or {A,B,C}.
Sections in the Chinese source: 背景与主要改动, 方法与本地协议. The example illustrates the mechanism.
Structured Decisions / Candidate Invariance: Order, Competitors, and Calibration
Core mechanism
Why can exact-set recovery fail even when every correct option outranks distractors?
Show answer and example
Answer: One correct choice can absorb most listwise mass, leaving another below threshold or the uniform baseline. Correct ranking differs from exact-set recovery, and shares are not independent probabilities. Fit deployment thresholds on validation only.
Illustrative example: Example: Two correct options rank first and second, but uneven shares cause thresholding to keep only one. A top-one hit misses the incomplete set.
Sections in the Chinese source: 背景与主要改动, 方法与本地协议. The example illustrates the mechanism.
Structured Decisions / Candidate Invariance: Order, Competitors, and Calibration
Method comparison
Why calibrate listwise shares differently from independent binary probabilities?
Show answer and example
Answer: Listwise shares depend on other options and sum to one; multiple binary probabilities may be simultaneously high. One threshold or calibration meaning confuses competition with correctness.
Illustrative example: Example: Adding another correct option can reduce one item's listwise share without reducing its binary probability.
Sections in the Chinese source: 背景与主要改动, 方法与本地协议. The example illustrates the mechanism.
Structured Decisions / Candidate Invariance: Order, Competitors, and Calibration
Implementation details
May the threshold be reselected from the test exact-set curve?
Show answer and example
Answer: No. Deployment selects thresholds on a validation grid and freezes them for test. A same-data threshold curve is retrospective. The predictor does not see the correct set.
Illustrative example: Example: Picking the best point after seeing test is a hindsight upper bound, not unseen-data performance.
Sections in the Chinese source: 背景与主要改动, 方法与本地协议. The example illustrates the mechanism.
Structured Decisions / Candidate Invariance: Order, Competitors, and Calibration
Basics
Why is request independence separate from structured output?
The Company a Question Keeps: Auditing Open Jev-Style Models for Retrieval
Show answer and example
Answer: Valid JSON or candidate labels guarantee format, not whether an unrelated question changes the original decision. The audit fixes evidence and target to measure interference from grouping requests.
Illustrative example: Example: Both questions can produce valid probabilities even if placing them together flips the first decision.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Structured Decisions / Candidate Invariance: Order, Competitors, and Calibration
Core mechanism
Why audit added questions, evidence and candidates separately?
The Company a Question Keeps: Auditing Open Jev-Style Models for Retrieval
Show answer and example
Answer: Adding a question keeps proposition and evidence fixed to test interference. New evidence legitimately changes context; new candidates change normalization. Combining them prevents attribution. For expansion, compare original conditional odds. Evaluator labels, scenario names and document identity never enter predictions.
Illustrative example: Example: An irrelevant extra question flipping a decision is concerning; new evidence may justify a flip, and a third choice can reduce shares without changing relative preference.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Structured Decisions / Candidate Invariance: Order, Competitors, and Calibration
Method comparison
How can candidate-expansion probability drops be distinguished from preference reversals?
The Company a Question Keeps: Auditing Open Jev-Style Models for Retrieval
Show answer and example
Answer: A new candidate enlarges the softmax denominator and can reduce old shares naturally. Compare conditional odds between original choices or strict binary flips to separate normalization from relative preference.
Illustrative example: Example: A loses share but still outranks B because C absorbs mass; A/B order has not flipped.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Structured Decisions / Candidate Invariance: Order, Competitors, and Calibration
Implementation details
Why is a binary probability of exactly 0.5 not a strict flip?
The Company a Question Keeps: Auditing Open Jev-Style Models for Retrieval
Show answer and example
Answer: Strict flips require centered probabilities on opposite sides, with product strictly negative. Equality is a separate tie. Assigning the boundary arbitrarily overstates flips and obscures precision effects.
Illustrative example: Example: 0.5 followed by a preference for yes is a boundary change, not a clear no-to-yes reversal.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Structured Decisions / Candidate Invariance: Order, Competitors, and Calibration
Basics
Must Prism-1's original probabilities stay numerically unchanged after a third option?
Prism-1: Position- and Rival-Independent Scoring for System One Models
Show answer and example
Answer: No. The denominator can increase and both shares can fall. Preserve their relative odds and order; the new option may win. The consistency constraint does not require ignoring additions.
Illustrative example: Example: Original 2:1 odds can become probabilities 0.4 and 0.2 after adding C, still 2:1.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Structured Decisions / Candidate Invariance: Order, Competitors, and Calibration
Core mechanism
Why combine isolated attention and equal starting positions in Prism-1?
Prism-1: Position- and Rival-Independent Scoring for System One Models
Show answer and example
Answer: The score may depend only on shared state, question and its own option. Isolation prevents cross-reading; equal positions remove order effects. Joint softmax may change the denominator but should preserve old odds. This does not suit cross-option choices such as 'none of the above' or prevent injection in shared state.
Illustrative example: Example: A new tool can win and lower old shares but insertion position alone should not swap the old tools' ranking.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Structured Decisions / Candidate Invariance: Order, Competitors, and Calibration
Method comparison
Why doesn't Prism-1's option independence suit 'none of the above'?
Prism-1: Position- and Rival-Independent Scoring for System One Models
Show answer and example
Answer: Independent scores see shared state, question and the option itself. 'None of the above' depends on the validity of other options and needs interaction, so imposing the same Luce constraint is inappropriate. This is a structural boundary, not a routine training error.
Illustrative example: Example: Adding a correct answer should invalidate 'none of the above', which an isolated option cannot determine.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Structured Decisions / Candidate Invariance: Order, Competitors, and Calibration
Implementation details
Can Prism-1 option blocks read each other or start at different positions?
Prism-1: Position- and Rival-Independent Scoring for System One Models
Show answer and example
Answer: No. Each reads shared context and attends causally within its own block; all start at the same position. Otherwise independent scoring breaks. This isolation does not block injection in shared context.
Illustrative example: Example: Inserting a longer option must not shift later options' position IDs.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Topic guide and reading order
Start with this question¶
Why can simply reordering options or adding an irrelevant candidate change the original preference?
Takeaway¶
Prism-1 starts from choice axioms: each option sees only shared state and itself, with isolated attention blocks and equal starting positions, before independent scores are normalized by softmax. The request audit separately adds questions, evidence, and candidate eligibility. This separates reliance on other questions, real new evidence, and effects from the normalization denominator.
Example¶
If A and B have exponentiated scores 2 and 1, adding C with score 5 drops A's probability from 2/3 to 2/8 naturally. A's odds against B should still be 2. If the old ranking reverses, investigate cross-option attention, positional changes, or schema leakage.
Common pitfall¶
Invariance does not guarantee accuracy or prevent C from winning. Tasks such as None of the Above may need cross-option semantics. Keep strict flips separate from ties and do not interpret an interval containing zero as proof of no effect. The local audit script does not generate the paper model's predictions.
Reading order¶
First read Prism-1 on independent option scores, block masks, and positions. Then read the three paired interventions in the request audit to learn how to avoid mistaking normalization for preference reversal.
- Prism-1: Position- and Rival-Independent Scoring for System One Models · Original paper
- The Company a Question Keeps: Auditing Open Jev-Style Models for Retrieval · Original paper
Check your understanding¶
Does A's absolute probability decreasing after adding C prove a violation of candidate independence?
Show suggested answer
No. Hold shared evidence fixed and compare old conditional probability A/(A+B) or odds A/B. Adding C to the denominator naturally lowers A's absolute probability. Distinguish new evidence from eligibility alone.