ULTRA-HSTU: Bend the scaling curve for large recommenders¶

Core semi-local attention, LBSL and mixture of transducers.

Paper information¶

Field	Details
Paper	Paper or original source
Institution	Meta.
First public release	2026-02-19 (arXiv v1)
Author code	No official author code linked by the paper as checked on 2026-08-09.
Local adapter or method key	`ultra-hstu`
Local reproduction code	`src/auto_research/reproductions/ultra_hstu/`

Paper method and evidence¶

At 16k behaviors and greater depth, HSTU attention and memory bottleneck scaling. Use local tokens and sparse landmarks, widen windows by layer with LBSL, and combine a mixture of transducers with attention truncation, mixed precision and dynamic topology.

flowchart LR
  A[Local tokens and landmarks] --> B[Widen attention by depth]
  B[Widen attention by depth] --> C[Mix transducers]

Original paper figure¶

Figure shows the scalable HSTU architecture. From the original paper; copyright remains with the authors.

Core equation¶

\[ \mathcal N_l(t)=\{t-w,\ldots,t\}\cup\mathcal B_l,\qquad w_l=w_0(l+1), \]

\[ h_{l+1}=h_l+\sum_e \operatorname{softmax}(r(h_l))_eT_e(\operatorname{Attn}_{\mathcal N_l}(h_l)). \]

Each layer attends only to its local window and block landmarks; LBSL widens the window with depth.

Reported results¶

Multiple 30-day A/B tests and full rollout to billions of daily users report consumption +4.11%, engagement +2% to +8%, topline +0.05%/+0.01%, training scaling efficiency 5× and inference 21×.

Local reproduction¶

Short-history transition/content control versus six layers, 8-token local windows and wider summaries: Hit@10 +8.33%, NDCG@10 -0.95%, fresh Hit@10 +50%, head share -13.38%. Seed-42 metrics. 48 events proxy 16k; no 18-layer production model, dynamic topology, FP8 kernel or private logs. No stable quality claim.

auto-research reproduce --paper ultra-hstu --dataset-dir data --seed 42