CHORD: Dynamically balance SFT and on-policy RL¶

Isolated RL update on a public candidate policy, not a paper-scale benchmark.

Paper information¶

Field	Details
Paper	CHORD (arXiv 2508.11408)
Institution	Alibaba Group / ModelScope.
First public release	2025-08-15
Author code	Trinity-RFT CHORD example.
Local adapter or method key	`chord`
Local reproduction code	`src/auto_research/post_training/`

Paper method and evidence¶

Separate SFT and RL stages can overfit expert data or forget it too early. Add expert SFT as an annealed auxiliary term during on-policy RL; token-level uncertainty weights smooth the transition from imitation to exploration.

flowchart LR
  E[Expert traces] --> S[Annealed SFT loss]
  R[On-policy rollouts] --> P[Group-RL loss]
  S --> U[Combine gradients]
  P --> U

Original paper figure¶

The original figure shows the dynamic training mixture. From the original paper; copyright remains with the authors.

Core equation¶

\[ \mathcal L=\lambda_t\mathcal L_{\rm SFT}+(1-\lambda_t)\mathcal L_{\rm RL},\qquad \lambda_t\downarrow\ \text{during training}. \]

Reported results¶

The paper reports gains over staged SFT+RL and static mixtures on multiple tasks. No production A/B.

Local reproduction¶

Use verified gold candidates as expert traces, anneal the SFT term, and add its gradient to the on-policy group-RL gradient.

auto-research post-train --algorithm chord --dataset gsm8k-candidate --maximum-examples 256 --steps 120 --seed 42

Fixed-seed summary. No learned LLM token-uncertainty model; the gold candidate is a deterministic expert proxy.