Long context has several axes: position, access, and state compression¶
简体中文 · English
Updated 2026-10-30 · 34 topic entries / 8 representative comparisons
Ask first how history is stored and then how the current token reads it. RoPE changes relative position relationships; Mamba writes history into recurrence; NSA splits access across paths; OpenAttention retains proxies for unread content; and TTR fits an input-conditioned compression. MERLIN keeps routing-key moments beside compressed segment states to change how segments are read without replacing their internal recurrent writes. A single supported-length number cannot rank these designs.
Mechanisms · Trade-offs · Cross-paper questions · Updates
How to read this card
The table compares representative methods. It is not a shared-benchmark leaderboard. Teaching examples are not measured results.
Mechanism comparison¶
Axes: Token-level KV, recurrent state, or input-conditioned compression coefficients · Whether the current query can revisit exact details · Whether unread history is omitted, summarized, or already folded into state · Costs in training, forward solving, or index access
| Paper and source | Mechanism | Benefits and trade-offs |
|---|---|---|
| RoFormer: Enhanced Transformer with Rotary Position Embedding · Original source Sections in the Chinese source: 背景与主要改动, 核心公式 |
Rotate two-dimensional Q/K subspaces so their dot product expresses relative position; compatible with full attention. | Encodes position rather than reducing history count or KV bandwidth. The local small-model rotation does not establish arbitrary length extrapolation. |
| Mamba: Linear-Time Sequence Modeling with Selective State Spaces · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Use token-conditioned step size and read/write vectors to fold history into bounded recurrent state. | Avoids retaining all token KV, but only information kept in state remains accessible. Local native PyTorch scan does not reproduce the fused kernel. |
| Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Use compressed blocks for overview, query-selected blocks for detail, and a sliding window for exact recency; gate their outputs together. | Different paths need design and training. A coarse overview may not recover details after a selection miss. Local trainability is not reported long-context throughput. |
| OpenAttention: One Percent Is Enough with Tail Proxies · Original source Sections in the Chinese source: 背景与主要改动, 核心公式 |
Cluster by query geometry, allocate exact-read budget by estimated proxy error, and include unread clusters as count-weighted mean proxies in softmax. | The tail is not zeroed, but within-cluster detail can disappear and ErrorGate relies on approximations. Only a single-head operator is diagnosed locally; full experiments were too expensive. |
| Rethinking Sequence Compression with Test-time Regression · Original source Sections in the Chinese source: 背景与主要改动, 核心公式 |
Treat token groups as regression observations and solve regularized least squares each forward pass for input-conditioned coefficients. | Neither weight training at inference nor free pooling. The dual solve has a T³ term and is attractive only when groups are much shorter than feature width. Local work is diagnostic. |
| Select-TTT: Query-Conditioned Selective Fast Weight Updates for In-Place Test-Time Training · Original source Sections in the Chinese source: 背景与主要改动, 核心公式 |
Aggregate top-k low-CE positions of query tokens, scale by T×softmax, and clip write multipliers. | Requires a real frozen head/RMSNorm and NTP-aligned features, not a local LLM or distributed kernel. |
| Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories · Original source Sections in the Chinese source: 背景与主要改动, 核心公式 |
Mix an external binary semantic reward with thresholded token edit similarity. | Implements LTI reward only, not CMS, dreaming, knowledge seeding, or LoRA; full Sleep training was too expensive and not reproduced. |
| Multi-State Linear Attention with Generalized Second-Order Routing · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 高成本完整实验 |
Keep segment routing-key means and raw second moments to recover query-aligned variance. Softmax the two paths separately and gate their external state readouts. | Adds segment-state and moment storage without reconstructing exact token softmax. The current unfinished segment needs external recurrence; full mid-training was not reproduced. |
Trade-off analysis¶
Expressible position is not accessible history¶
RoPE defines how QK carries relative position, but full attention still processes visible history. Mamba changes storage and propagation, replacing per-token access with state. Position extrapolation failure and forgetting a detail in state are different; extra state alone cannot explain the former and changing positions cannot fix the latter. Identify whether the task needs exact lookup or persistent state updates.
Sources: RoFormer: Enhanced Transformer with Rotary Position Embedding · Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Can compressed history still expose details?¶
NSA's compressed branch sits beside query-conditioned exact block reads. OpenAttention spends budget on exact rows and leaves the rest in normalization as log-count mean proxies. TTR maps whole groups to regression coefficients with no per-item invertibility guarantee. Trace where missing information goes rather than counting output vectors. Local formulas or short runs do not rank long-text capability.
Sources: Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention · OpenAttention: One Percent Is Enough with Tail Proxies · Rethinking Sequence Compression with Test-time Regression
October 29: new methods and evidence boundaries¶
Select-TTT changes how strongly each context chunk writes to fast weights based on current query token probability mass; apply-before-write preserves temporal order. Clipping can leave the total below the chunk count and must not be renormalized to fake conservation. Sleep separates active learning from offline consolidation; locally only LTI reward runs. Its full method trains added parameters, so the reward operator does not establish memory written into the model.
Sources: RoFormer: Enhanced Transformer with Rotary Position Embedding · Select-TTT: Query-Conditioned Selective Fast Weight Updates for In-Place Test-Time Training · Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
Write state first, then choose which segments to read¶
Mamba's selective recurrence controls what enters a state; MERLIN assumes compressed segments and readouts already exist and routes the current query among them. Equal key means can hide a few strong matches. Raw second moments recover aligned variance but not missing within-segment detail. Eq. 9 normalizes means and variances separately before mixing, unlike mixing logits then softmax. Equal-length segments and causal visibility are required.
Sources: Multi-State Linear Attention with Generalized Second-Order Routing · Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Cross-paper questions¶
Try answering before expanding the explanation. Each question relates at least two papers.
Why are changing RoPE and switching to recurrence not the same fix for a forgotten early one-off identifier?¶
Show answer and example
Answer: RoPE changes position in QK, not access to original evidence. Mamba changes selective state writing; if it omits the identifier, no original KV remains to query. Diagnose position distortion, invisibility, or compression loss first, then decide whether to use detail access such as NSA.
Illustrative example: An order ID appears once on the contract cover. Correct relative positions do not ensure it is read, and bounded state does not retain every ID.
Sources: RoFormer: Enhanced Transformer with Rotary Position Embedding · Mamba: Linear-Time Sequence Modeling with Selective State Spaces · Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
How do selected NSA blocks and OpenAttention unread-cluster proxies handle omissions differently?¶
Show answer and example
Answer: NSA gates overview, selected blocks, and recent-window outputs; OpenAttention includes unread clusters as count-corrected proxies in the same attention normalization as exact rows. Multiple coverage paths and tail mass/value approximation can both lose within-group details.
Illustrative example: One outlier among 100 records can be diluted by either a coarse summary or a mean; test whether selection routes it to an exact read.
Sources: Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention · OpenAttention: One Percent Is Enough with Tail Proxies
Do TTR input-wise fitting and Mamba per-token state updates mean the deployment model trains its weights?¶
Show answer and example
Answer: No. TTR solves coefficients in the current group's forward pass using outer-trained targets and regularization. Mamba uses fixed parameters and current tokens to update state. These are input-dependent computation or runtime state, not optimizer updates, and have different solve versus recurrence costs.
Illustrative example: One frozen model can produce different states and coefficients for two texts without writing a new checkpoint.
Sources: Rethinking Sequence Compression with Test-time Regression · Mamba: Linear-Time Sequence Modeling with Selective State Spaces
What actual execution is evidenced by query-conditioned write coefficients versus Sleep reward changes?¶
Show answer and example
Answer: Local Select-TTT updates a small fast-weight matrix, allowing pre-write outputs and final state checks. Local Sleep returns a reward without an optimizer or parameter expansion. Neither establishes long-context capability, and reward is not long-term retention.
Illustrative example: Updating a memory table and scoring a continuation 0.9 are different evidence.
Sources: RoFormer: Enhanced Transformer with Rotary Position Embedding · Select-TTT: Query-Conditioned Selective Fast Weight Updates for In-Place Test-Time Training · Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
Does adding a second moment turn MERLIN's bounded Mamba-like state into full attention?¶
Show answer and example
Answer: No. The moment estimates alignment variance between the query and each segment's routing keys to combine readouts. Token values remain compressed and are not individually normalized; an added access cue cannot reverse compression.
Illustrative example: Means of 1 do not distinguish a segment of all ones from a segment of −1 and 3. Variance distinguishes them but cannot reconstruct every original record.
Sources: Multi-State Linear Attention with Generalized Second-Order Routing · Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Update log¶
Updates record new approaches, changed trade-offs, and follow-up evidence.
2026-10-30 · Reviewed MERLIN Eqs. 7–9: distinguish raw second moments, alignment variance, and two-softmax gated readout from Mamba recurrence; document completed causal segments, the external current segment, and the lack of exact attention.
Affected: Multi-State Linear Attention with Generalized Second-Order Routing · Mamba: Linear-Time Sequence Modeling with Selective State Spaces
2026-10-29 · Select-TTT scales fast-weight writes by query token probability mass and preserves apply-before-write order. Do not renormalize the clipped total to the chunk count. Sleep uses distinct active and consolidation stages; only its LTI reward runs here. Unlike frozen-parameter state updates, full Sleep trains extensions, which a reward operator cannot establish.
Affected: Select-TTT: Query-Conditioned Selective Fast Weight Updates for In-Place Test-Time Training · Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
2026-10-27 · Initial version: compared position, selective state, three sparse read paths, tail proxies, and forward regression compression by how they preserve and recover history.
Covers 31 topic entries; see the full list below.
All 34 topic entries
The full catalog follows. The main comparison covers the representative papers listed above.
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
- Convolution for Large Language Models
- DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling
- Fractional State Space Transition for Long Sequence Modeling
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
- Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection
- Hymba: A Hybrid-head Architecture for Small Language Models
- Kronecker Delta Rule for Long Context Linear Attention
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Open-world AI: a Tale of Three Requirements
- MiniMax Sparse Attention
- MoBA: Mixture of Block Attention for Long-Context LLMs
- Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable
- MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution
- Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory
- Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
- Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels
- On-Demand Attention: Efficient Long-Context Decoding with Learned Recall
- Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension
- OpenAttention: One Percent Is Enough with Tail Proxies
- Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers
- Role-aware Heuristic Episodic Attention for Conversational LLMs
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
- Stratified Sparse Attention: Reducing the Sparse-to-Dense Gap at Extreme Sparsity
- Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformers
- TransMem: Transforming Hidden States into Memory for Large Language Models
- Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling
- Learning What to Remember: Test-Time Training via Context Distillation
- Rethinking Sequence Compression with Test-time Regression
- Select-TTT: Query-Conditioned Selective Fast Weight Updates for In-Place Test-Time Training
- Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
- Multi-State Linear Attention with Generalized Second-Order Routing