TTRL: online reinforcement learning on unlabeled test sets¶

Fidelity: core mechanism. The deterministic mini-suite does not substitute for the full paper benchmark.

Paper information¶

Field	Details
Paper	TTRL: online reinforcement learning on unlabeled test sets (arXiv 2504.16084)
Organization	PRIME-RL author team
First public release	2025-04-22 (arXiv v1)
Authors' code	Public implementation
Adapter / method	`ttrl`
Local reproduction code	`src/auto_research/post_training/`

Original paper¶

Motivation and method¶

Sample several answers for the same test question, use the majority answer as a pseudo-label, and update immediately without gold labels.

flowchart LR
 A["Public input or history"] --> B["TTRL training and execution loop"]
 B --> C["Paper-specific state or objective"]
 C --> D["Matched-budget evaluation"]

Key figure from the paper¶

Figure 2: Overview of the paper's method and key components. From the original paper; copyright remains with the authors.

Core equations¶

\[ \hat y=\operatorname{mode}\{y_k\}_{k=1}^K,\quad r_k=\mathbf1[y_k=\hat y],\quad\theta' = \theta+\eta\nabla J_{GRPO}(r). \]

Paper results¶

The paper reports test-time gains across reasoning benchmarks and model sizes. It reports no production A/B, so there are no inferred online numbers.

Local reproduction¶

Arithmetic candidate suite, 120 steps, 256 examples, seed 42: accuracy 0.2344 → 0.2344 (+0.00%). Reward, KL, length, and candidate budgets match.

auto-research post-train --algorithm ttrl --dataset arithmetic-smoke --maximum-examples 256 --steps 120 --seed 42
auto-research evolve --model post-training --dataset arithmetic-smoke --direction "组合 ttrl 与已安装论文算子" --generations 2 --population 4

See the fixed metrics.

Reproduction boundaries¶

The local run checks the paper-specific objective, state updates, and fair budgets. It does not reproduce large models, multi-GPU RL, private environments, real websites, or the full benchmark. This is mechanism validation, not reproduction of the original tables.