CoRT: token-level rubric credit through counterfactual replay¶

简体中文

Fidelity: Runs the paper's isolatable state and update mechanisms on common public data or a deterministic mini-suite. This does not reproduce the original large model.

Paper information¶

Field	Details
Paper	CoRT: token-level rubric credit through counterfactual replay (arXiv 2607.25659)
Organization	ByteDance internship / academic author team
First public release	2026-07-28
Authors' code	No official public repository was found as of 2026-07-29.
Adapter / method	`cort`
Local reproduction code	`src/auto_research/post_training/`

Original paper¶

Motivation and method¶

Replay the same response with a rubric and with criteria removed. Token likelihood differences redistribute GRPO's response-level advantage.

flowchart LR
    Y["Same response"] --> R["Rubric-conditioned replay"]
    Y --> C["Criteria-free replay"]
    R --> D["Token likelihood contrast"]
    C --> D
    D --> U["Reweighted GRPO"]

Key figure from the paper¶

Figure 2: Overview of the paper's method and key components. From the original paper; copyright remains with the authors.

Core equations¶

\[ w_t=\operatorname{Norm}\!\left(\log\pi_\theta(y_t\mid x,r)-\log\pi_\theta(y_t\mid x)\right),\quad \mathcal L=-\sum_t w_tA\log\pi_\theta(y_t). \]

Paper results¶

Most paired experiments across models and reward granularities beat response-level GRPO, averaging +4.4 percentage points, without an extra token scorer. No production online A/B is reported; the evidence is offline or environment-based.

Local reproduction¶

On the GSM8K candidate policy, run rubric and criteria-free counterfactual replays, normalize token weights, and update with signed advantages.

Metric	Untrained policy	CoRT
Accuracy	0.1719	0.8906
Mean reward	0.3124	0.8908
KL(reference)	0.0000	0.0201

auto-research post-train --algorithm cort --dataset gsm8k-candidate --maximum-examples 256 --steps 120 --seed 42

See the fixed-seed metrics.

Reproduction boundaries¶

Candidate-action features stand in for token sequences. No open-ended generation, rubric judge, or large-model parameter update is included.