TokenMixer-Large: scaling token mixing for industrial fine ranking¶
Fidelity: full core path. The backbone, interval residuals, and intermediate auxiliary loss are trained, at reduced scale.
Paper information¶
| Field | Details |
|---|---|
| Paper | arXiv 2602.06563 |
| Organization | ByteDance |
| First public release | 2026-02-06 (arXiv v1) |
| Authors' code | The paper did not supply official or author code when checked on 2026-08-09. |
| Adapter / method | tokenmixer-large |
| Local reproduction code | src/auto_research/reproductions/tokenmixer_large/ |
Original paper¶
Motivation and method¶
TokenMixer-Large applies parameter-free mixing to candidate and multidomain feature tokens, then head-wise SwiGLU. After reverting the mix it applies token-wise SwiGLU. Interval residuals and intermediate supervision stabilize scaling of deep fine-ranking models.
flowchart LR
A["Feature tokens"] --> B["Mixing"]
B --> C["Head-wise SwiGLU"]
C --> D["Reverting"]
D --> E["Token-wise SwiGLU"]
E --> F["Interval residual + aux head"]Key figure from the paper¶
Figure 1: Core architecture, modules, and their connections. From the original paper; copyright remains with the authors.
Core equations¶
Paper results¶
E-commerce orders +1.66%, preview-payment GMV per person +2.98%, ads ADSS +2.0%, and livestream revenue +1.4%.
Local reproduction¶
On MovieLens 100K, TokenMixer-Large and rankmixer_dense share dimensions, layer count, negative sampling, and training steps.
Local comparison: Against same-protocol RankMixer dense, TokenMixer-Large increases NDCG@10 from 0.00385 to 0.00414, or +7.54% relative; see
metrics/movielens-100k-seed42.json.