IcePop: Two-sided train–inference mismatch masking¶

Candidate-policy RL mechanism, not trillion-parameter MoE.

Paper information¶

Field	Details
Paper	Paper or original source
Institution	Ant Group / Inclusion AI.
First public release	2025-10-21
Author code	No standalone IcePop algorithm source found or released.
Local adapter or method key	`icepop`
Local reproduction code	`src/auto_research/post_training/`

Paper method and evidence¶

MoE routing amplifies numerical differences between training and rollout engines. One-sided TIS may retain very small mismatched ratios. Mask ratios outside a fixed two-sided interval and retain original corrections inside.

flowchart LR
  A[Compare engine probabilities] --> B[Apply two-sided ratio mask]
  B[Apply two-sided ratio mask] --> C[Keep in-range correction and PPO clipping]

Original paper figure¶

The original figure shows mismatch handling. From the original paper; copyright remains with the authors.

Core equation¶

\[ \rho_t^{\rm TI}=\frac{\pi_{\rm train}(a_t\mid s_t)}{\pi_{\rm rollout}(a_t\mid s_t)},\qquad m_t=\mathbf1[c_{\rm low}\le\rho_t^{\rm TI}\le c_{\rm high}],\qquad \mathcal L=-\mathbb E[m_t\rho_t^{\rm TI}r_t^{\rm policy}A_t]. \]

Reported results¶

Ring-1T uses IcePop with C3PO++ and ASystem for trillion-parameter MoE RL and reports AIME 2025 and other scores, without an isolated production IcePop A/B.

Local reproduction¶

Same candidate policy and mismatch simulation as TIS. Use [0.5, 5.0] and zero gradients for out-of-range actions while preserving PPO ratio/clipping against old rollout policy. Metrics. No real MoE routing or async cluster.

auto-research post-train --algorithm icepop --dataset gsm8k-candidate --maximum-examples 256 --steps 120 --seed 42