RLCR: Train confidence with proper scoring rules

简体中文

Scoring-rule mechanism, not the paper's LLM rollouts, RL stack, or confidence-weighted scaling.

Paper information

Field	Details
Paper	arXiv 2507.16806
Institution	Massachusetts Institute of Technology (first author).
First public release	2025-07-22 (arXiv v1)
Author code	damanimehul/RLCR.
Local adapter or method key	system-one:rival-brier, system-one:rival-hybrid
Local reproduction code	src/auto_research/system_one/local.py

Paper method and evidence

Binary correctness alone can reward indiscriminate confidence. RLCR predicts both an answer and its confidence and adds a bounded proper scoring rule so honest probabilities maximize expected reward. The local baseline minimizes multiclass Brier over the full candidate distribution.

Original paper figure

Figure 11 shows the original training curves. From the original paper; copyright remains with the authors.

Core equation

\[ \mathcal{L}_{Brier}=\sum_k(p_k-y_k)^2, \qquad \mathcal{L}_{hybrid}=\tfrac12(\mathcal{L}_{CE}+\mathcal{L}_{Brier}). \]

Gradients propagate through the full softmax rather than treating confidence as a display-only field.

Reported results

The authors report better calibration without losing accuracy across in-domain and out-of-domain tasks. The local results compare objectives only within a shared Banking77 protocol.

Local reproduction

The local implementation verifies the proper-scoring objective without the authors' checkpoint or RL rollouts.