Structured decisions by topic¶

Retrieval judgments and reranking evaluation¶

Multimodal Reranking Is a System One Decision: Instead of generating a ranked list or training a separate pointwise reranker, the model scores each candidate by reading the probability of its letter label at the first answer position. The readout follows setwise and FIRST-style methods; the paper tests how this approach works zero-shot on images and visual documents, and whether text-only decision tuning transfers to visual reranking. Bounded component only; full experiments are not reproduced.
Jev-as-a-Judge for Brazilian Portuguese Information Retrieval: Relevance Assessment and Reranking: On Quati it separately uses an independent Noul probability that each document directly answers the query, gathers every batch, and sorts the results. Candidates do not compete in a single Choice, probabilities are not mixed with BM25 scores, and there is no trained calibrator. Evaluation protocol interface; not a model capability result.

TypeSafe Jev: Choice, Score and Noul questions share state input and return machine-readable probabilities.

Prism-1: A softmax combines independent scores. Softmax alone cannot guarantee this without attention isolation and position reset. Bounded component only; full experiments are not reproduced.
GLiClass: This avoids unstable parsing of generative LLM outputs and separate cross-encoder inference on each text–label pair. The repository retains the defining mechanisms: candidate content affects scores, and one normalization produces a distribution over all candidates.
Bespoke Nimble: Trains typed-schema logits on Qwen3.5-9B; checkpoint, data and recipe are public.
NanoJev: Trains Choice, Boolean and Score decision heads on Qwen3-0.6B.
Laya: Trains a typed decision model and publishes its temperature-calibration path.

SemIf: Reads candidate logits after a shared-state prefill; an engineering approach, not a trained reproduction of Jev.
openjev-sglang: Checks GPU serving with parallel constrained decoding in SGLang.
See the community implementation snapshot for other lightweight wrappers.

DiffusionGemma, vLLM, and other routes: Explore non-autoregressive parallel decisions and evaluate separately from trained decision heads.

When Multiple Choices Are Correct: Distinguish normalized categorical shares from independently queried binary correctness probabilities; the former are not calibrated binary confidence. The paper studies Jev, Luna and Gemini. Bounded component only; full experiments are not reproduced.
RLCR: RLCR predicts both an answer and its confidence and adds a bounded proper scoring rule so honest probabilities maximize expected reward. The local baseline minimizes multiclass Brier over the full candidate distribution.
Calibration-Aware RL: The paper adjusts decision-token probabilities directly to reduce ECE without sacrificing accuracy. The repository reports accuracy alongside Brier score and ECE, fits temperature only on validation, and forbids test-set tuning.

The unified protocol reports coverage and selective accuracy, but lacks a public comparable abstention benchmark spanning multiple difficulty levels. See the P1 gap.
jev-ood-calibration is a candidate comparator for future OOD and coverage–risk evaluation.

Score and Noul are in the typed contract and local provider. A public data protocol for ordinal scoring, Boolean risk decisions and cost-sensitive routing is still needed.