MS-GLA: Multi-Temporal Gated Linear Attention

简体中文

L1 architecture with real A100 forward/backward. Multiscale decomposition, causal hold, and input-dependent fusion are implemented, not the paper's Flash kernel or matched-parameter training.

Paper information

Field	Value
Paper	MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution
Affiliation	First author: International Institute of Information Technology Hyderabad
First public release	2026-09-28 (arXiv v1)
Authors' code	prasoondev/msgla
Adapter / method	ms-gla
Local code	foundation_latest_20260930.py

Original paper

Background and contribution

One time resolution can bottleneck linear attention. Allocate heads over scales, run nonoverlapping masked-average pooling and separate GLA branches, then align back with causal hold and fuse through input-dependent softmax routing.

flowchart LR
 X[Token states] --> P1[Scale 1] --> G1[GLA]
 X --> P2[Scale 2 pooling] --> G2[GLA]
 X --> P4[Scale 4 pooling] --> G4[GLA]
 G1 --> F[Causal alignment and softmax fusion]
 G2 --> F
 G4 --> F

Key figure from the paper

Branches at different resolutions and their fusion, from the paper. Copyright belongs to the authors.

Core equation and evidence

A scale-\(s\) masked block average is available only when its final token arrives, avoiding future leakage. \(\alpha_{t,s}=\operatorname{softmax}_s(W_f h_t)\) and output \(\sum_s\alpha_{t,s}\tilde o_{t,s}\). Language modeling and throughput belong to the paper. Local metrics check pooling, causal hold, normalized routes, and finite gradients.

Local reproduction

MultiScaleGLA is a compact differentiable PyTorch layer. The aggregate experiment and sanitized NVIDIA A100 receipt verify CUDA execution only.

Reproduction limits

The recurrent reference favors correctness, not throughput. No Triton/Flash replication, paper-scale parameter budget, or executed Evolve integration. A registry name is not a runnable search operator.