SPIRAL: Reasoning through Zero-Sum Language-Game Self-Play

简体中文

Core mechanism evidence. Paper and local evaluations are separate.

Paper information

Field	Value
Paper	SPIRAL (arXiv 2506.24119)
Affiliations	Apple / academic collaborators
First public release	2025-06-30 (arXiv v1)
Authors' code	Author code.
Adapter / method	spiral
Local code	src/auto_research/post_training/

Original paper

Background and contribution

The same model plays problem setter and solver. Automatically scored zero-sum multiround games create a progressively harder curriculum.

flowchart LR
  M[Shared model] --> P[Propose challenge] --> S[Solve challenge] --> V[Zero-sum verification] --> M

Key figure from the paper

Original Figure 3: SPIRAL. From the paper; copyright belongs to the authors. Not a local result.

Core mechanism

\[ \max_{\pi_A}\min_{\pi_B}\mathbb E_{\tau\sim(\pi_A,\pi_B)}[R(\tau)],\quad A_A=-A_B. \]

Paper results

Self-play alone improves reasoning benchmarks and transfers to small models in the paper. No production A/B.

Local reproduction

Arithmetic candidate suite, 120 steps, 256 examples, seed 42: 0.2344→0.3438 (+46.67%) with matched reward, KL, length, and candidate budgets. Metrics.

auto-research post-train --algorithm spiral --dataset arithmetic-smoke --maximum-examples 256 --steps 120 --seed 42
auto-research evolve --model post-training --dataset arithmetic-smoke --direction "Combine spiral with installed paper operators" --generations 2 --population 4

Reproduction limits

A mechanism and fixed-budget check, not the paper's large models, distributed RL, real web/private environments, or full benchmarks.