Decisions on Demand: Selective Reasoning, Transfer, and Feedback¶
简体中文 · English
Structured Decisions / 16 questions · 4 papers
Try answering each question before opening the explanation. The bank grows as papers and questions are added.
Read this topic's comparisons, analysis, and cross-paper Q&A
16 questions · 4 papers
Structured Decisions / Decisions on Demand: Selective Reasoning, Transfer, and Feedback
Basics
Why is the Jev capability paper an evaluation rather than a new architecture?
Show answer and example
Answer: It tests knowledge, reasoning and multilingual behavior under finite candidates. The contribution is protocol and capability boundaries, not a new training objective. An evaluator neither implements nor improves the model.
Illustrative example: Example: Converting public questions to four choices and scoring them measures performance; it is not a reasoning module.
Sections in the Chinese source: 背景与主要改动, 数据与评测契约. The example illustrates the mechanism.
Structured Decisions / Decisions on Demand: Selective Reasoning, Transfer, and Feedback
Core mechanism
Why not combine vendor and local Jev capability numbers into one leaderboard?
Show answer and example
Answer: Different prompts, backends and data protocols confound capability. Fix public prompts, candidates and one decision; join labels only in scoring, recording language and missing coverage. This is an audit, not training, and one task class cannot represent all reasoning.
Illustrative example: Example: Vendor math under another prompt and local knowledge under a fixed interface can be listed separately, not used to declare an overall winner.
Sections in the Chinese source: 背景与主要改动, 数据与评测契约. The example illustrates the mechanism.
Structured Decisions / Decisions on Demand: Selective Reasoning, Transfer, and Feedback
Method comparison
Do strong knowledge and weak math scores prove the model cannot reason?
Show answer and example
Answer: No. Evidence is limited to tested tasks, prompts and choice interface, possibly with language/coverage effects. Report measured tasks and gaps separately instead of generalizing to every setting.
Illustrative example: Example: Failure on one arithmetic format does not establish failure in tool planning or all mathematics.
Sections in the Chinese source: 背景与主要改动, 数据与评测契约. The example illustrates the mechanism.
Structured Decisions / Decisions on Demand: Selective Reasoning, Transfer, and Feedback
Implementation details
Why submit question and candidates once in the Jev capability evaluation?
Show answer and example
Answer: One public decision prevents answer leakage or retry assistance. The scorer reads gold only afterward. Record vendor protocols separately and disclose missing items rather than silently removing them from the denominator.
Illustrative example: Example: Strip the answer field before calling the predictor; score through a separate stage.
Sections in the Chinese source: 背景与主要改动, 数据与评测契约. The example illustrates the mechanism.
Structured Decisions / Decisions on Demand: Selective Reasoning, Transfer, and Feedback
Basics
Does Jev-DLC continual adaptation need to retain complete histories?
Jev-DLC: Decision-Level Continual Learning from Typed Feedback
Show answer and example
Answer: No. A frozen scorer has two compact states: semantic counts/direction for the conservative channel and a current correction direction for the responsive channel. Feedback enters state, not an ever-growing conversation prefix.
Illustrative example: Example: Repeated routing corrections update counts and directions rather than append every old ticket.
Sections in the Chinese source: 背景与主要改动, 双通道、PA-I与可靠性. The example illustrates the mechanism.
Structured Decisions / Decisions on Demand: Selective Reasoning, Transfer, and Feedback
Core mechanism
Why assess Jev-DLC reliability before updates and apply feedback only later?
Jev-DLC: Decision-Level Continual Learning from Typed Feedback
Show answer and example
Answer: Reliability measures prediction before feedback, not after learning the answer. The conservative state tracks repeated semantic experience, the responsive state tracks current correction, and reliability mixes them. Feedback must bind a committed decision; no forgery/replay or reading current gold before prediction.
Illustrative example: Example: A user undo can update both channels for the next request, not retroactively turn this choice into a success.
Sections in the Chinese source: 背景与主要改动, 双通道、PA-I与可靠性. The example illustrates the mechanism.
Structured Decisions / Decisions on Demand: Selective Reasoning, Transfer, and Feedback
Method comparison
Why separate conservative and responsive Jev-DLC channels?
Jev-DLC: Decision-Level Continual Learning from Typed Feedback
Show answer and example
Answer: The conservative one builds a stable prior from repeated behavior; the responsive one tracks recent corrections. Reliability mixing balances long-term patterns and change; one state struggles to express both.
Illustrative example: Example: A long-standing preference for A and temporary switch to B can be recorded separately.
Sections in the Chinese source: 背景与主要改动, 双通道、PA-I与可靠性. The example illustrates the mechanism.
Structured Decisions / Decisions on Demand: Selective Reasoning, Transfer, and Feedback
Implementation details
Why reject feedback on forged, replayed or stale Jev-DLC Decisions?
Jev-DLC: Decision-Level Continual Learning from Typed Feedback
Show answer and example
Answer: Feedback must refer to a committed decision from this instance; otherwise arbitrary targets or duplicated experience can alter state. Release the current public cache after update. Adaptation never overrides hard safety masks.
Illustrative example: Example: Sending the same undo event twice must not count as two corrections.
Sections in the Chinese source: 背景与主要改动, 双通道、PA-I与可靠性. The example illustrates the mechanism.
Structured Decisions / Decisions on Demand: Selective Reasoning, Transfer, and Feedback
Basics
Why report accuracy, calibration and risk–coverage in regional-language transfer?
Show answer and example
Answer: Accuracy measures aggregate correctness; calibration matches predicted probabilities to frequencies; risk–coverage asks whether confidence filtering retains reliable examples. Each informs whether a low-cost model and stronger fallback make sense.
Illustrative example: Example: Models with the same accuracy may differ greatly in how concentrated errors are at low confidence.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Structured Decisions / Decisions on Demand: Selective Reasoning, Transfer, and Feedback
Core mechanism
Why can a well-calibrated model still route errors poorly by low confidence?
Show answer and example
Answer: Calibration checks probabilities against aggregate correctness; routing needs errors concentrated at lower confidence than correct cases. Select fraction on validation macro-F1 and freeze for test. Parallel languages share sentence identities; cluster comparisons by sentence rather than counting independent duplicates.
Illustrative example: Example: Mean confidence matches accuracy, but errors are spread across confidence levels. Routing the lowest-confidence cases may not save compute.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Structured Decisions / Decisions on Demand: Selective Reasoning, Transfer, and Feedback
Method comparison
How does routing low-confidence examples differ from translating all examples first?
Show answer and example
Answer: A cascade allocates extra compute to selected cases; translation changes language representation for every input. Translation gains do not show error-detection skill. Select and freeze routing fraction on validation.
Illustrative example: Example: Escalating only difficult examples need not translate them. Translating everything into Indonesian changes even easy inputs.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Structured Decisions / Decisions on Demand: Selective Reasoning, Transfer, and Feedback
Implementation details
What data may choose cascade fraction and local temperature?
Show answer and example
Answer: Validation predictions only: select fraction by the specified grid and macro-F1, and temperature by validation NLL. Freeze before test. Temperature calibration is a local extension, not the authors' training step.
Illustrative example: Example: If test suggests escalating more, don't revise the fraction and report the same score as independent.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Structured Decisions / Decisions on Demand: Selective Reasoning, Transfer, and Feedback
Basics
Why not generate equally long reasoning for all Selective Reasoning questions?
Selective Reasoning for Typed Decision Models: When to Start, Read, and Stop
Show answer and example
Answer: Direct readout suffices for some choices; others need extra steps. Gate reasoning by expected benefit and read answers mid-generation to allocate compute where it is needed.
Illustrative example: Example: Return a simple intent immediately, but reason through multi-step arithmetic and stop when sufficiently certain.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Structured Decisions / Decisions on Demand: Selective Reasoning, Transfer, and Feedback
Core mechanism
Why separate starting reasoning, triggering readout and accepting an answer?
Selective Reasoning for Typed Decision Models: When to Start, Read, and Stop
Show answer and example
Answer: The launch gate estimates benefit, the trigger chooses a natural prefix, and post-read normalized entropy decides acceptance/cancellation. They act on different states. Mark forced returns at budget exhaustion separately from threshold success; fitting cannot see evaluation gold.
Illustrative example: Example: Reason until a readable point, continue if the readout remains uncertain, and mark budget-limited uncertain output as forced rather than confident.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Structured Decisions / Decisions on Demand: Selective Reasoning, Transfer, and Feedback
Method comparison
How does reading at natural generation states differ from fixed-token polling?
Selective Reasoning for Typed Decision Models: When to Start, Read, and Stop
Show answer and example
Answer: The trigger seeks positions suitable for explicit readout; a rate limit only bounds frequency. A shared reader and entropy rule still decide reliability. Triggering is not proof of correctness.
Illustrative example: Example: Read after an intermediate conclusion rather than mid-equation, but still inspect candidate probabilities.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Structured Decisions / Decisions on Demand: Selective Reasoning, Transfer, and Feedback
Implementation details
Why normalize entropy by log K when candidate count changes?
Selective Reasoning for Typed Decision Models: When to Start, Read, and Stop
Show answer and example
Answer: Maximum entropy changes with K, so H/log K makes certainty thresholds comparable. Distinguish cancellation from budget exhaustion. Freeze gate normalization statistics and weights before evaluation, not by test feedback.
Illustrative example: Example: Four and ten choices have different raw entropy ranges; one raw threshold changes the stopping rule.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Topic guide and reading order
Start with this question¶
Which requests deserve additional reasoning, and which feedback should affect the next decision?
Takeaway¶
Selective Reasoning separates starting reasoning, reading a candidate distribution, and accepting and stopping. A shared reader works from both an empty prefix and intermediate reasoning. Jev-DLC keeps bounded action memory and correction directions over a frozen scorer, applying accept/revoke feedback to later requests. Transfer also tests changes in language and label budget.
Example¶
A user's change from tool A to B can inform similar future requests, but cannot turn the previous prediction into a success retroactively. On a harder item, read the distribution mid-reasoning and apply a preregistered stop rule. Record budget exhaustion separately from confident early stopping.
Common pitfall¶
Low entropy is concentration, not correctness. Freeze gates and thresholds before evaluation. Keep feedback chronological and never use future labels for transfer or ongoing adaptation. Local evidence consists of recomputed published statistics or interface diagnostics on random inputs, not high-cost full-model runs.
Reading order¶
Start with Selective Reasoning's start/read/stop controls for one request. Then read Jev-DLC's bounded feedback state and temporal protocol for adaptation across requests. Finally check the limits of public capability and cross-language transfer.
- Selective Reasoning for Typed Decision Models: When to Start, Read, and Stop · Original paper
- Jev-DLC: Decision-Level Continual Learning from Typed Feedback · Original paper
Check your understanding¶
Can answering a question again immediately after its correct answer is fed back count as online continual-learning success?
Show suggested answer
No. The current target has been seen, so the new answer is not an independent prediction. Log the initial decision and feedback, then evaluate the updated state only on future unrevealed requests, against a no-feedback or fixed-policy baseline on the same sequence.