Long context: attention, recurrent state, and memory¶
简体中文 · English
Foundation models / 136 questions · 34 papers
Try answering each question before opening the explanation. The bank grows as papers and questions are added.
Read this topic's comparisons, analysis, and cross-paper Q&A
136 questions · 34 papers
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why does ALiBi address training short and testing long?
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Show answer and example
Answer: Position mechanisms fitted only to training lengths can fail at greater distances. ALiBi computes a distance penalty directly in logits. The positional rule extrapolates; long-context accuracy is empirical.
Illustrative example: After training on 512 words, compute distances for word 800 without looking up a learned 800th position.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Without position vectors, how does ALiBi tell how far apart tokens are?
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Show answer and example
Answer: It subtracts a distance-proportional bias from each head's attention logits. Distant tokens need a stronger content match, and slopes vary by head. The rule extrapolates without a fixed-length table, but that does not guarantee arbitrary-length retrieval.
Illustrative example: Equal content scores at distances 2 and 20 favor the closer token; a distant answer can still win with a stronger match.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
Where do linear bias and position embeddings act?
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Show answer and example
Answer: Position embeddings modify token inputs before Q/K/V projections. ALiBi does not learn position vectors; it modifies per-head matching logits directly. These are distinct information paths.
Illustrative example: The same content vectors can have different logit penalties at different distances.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Can ALiBi's distance penalty replace causal masking?
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Show answer and example
Answer: No. Bias changes preferences among visible positions, while the mask excludes future ones. Broadcast each head's slope and keep causality. Small local models do not establish the paper's large-scale gain.
Illustrative example: Token 3 cannot read token 4 even though their distance is one.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
What does data-free omit in Autonomy-of-Heads?
Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
Show answer and example
Answer: It uses frozen Q/K geometry rather than calibration examples and runtime gating to choose sparsity. This does not remove model pretraining or post-sparsification evaluation.
Illustrative example: Assign head types from weights before deployment, then check actual answers.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
How can Autonomy-of-Heads assign sparse patterns without calibration examples?
Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
Show answer and example
Answer: It analyzes frozen Q/K projections and uses spectral effective rank to characterize potential matching geometry, separating retrieval-oriented from streaming heads. No sample-based calibration or runtime selector is needed, but real tasks must validate the pattern.
Illustrative example: Two heads with equal budgets can emphasize very different numbers of directions; assign patterns accordingly and test evidence retention.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How does weight-based head assignment differ from dynamic attention selection?
Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
Show answer and example
Answer: Weight geometry gives stable assignments and avoids per-input selector work. Dynamic methods adapt to content but pay selection cost. Neither accuracy follows from calibration cost alone.
Illustrative example: A fixed retrieval head always reads far away, while a dynamic gate may allocate different ranges for short and long prompts.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Can singular values be used as probabilities directly for spectral entropy?
Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
Show answer and example
Answer: Normalize them into a distribution, or rescaling the matrix changes entropy's meaning. Handle degenerate cases. Local mini-suite checks data flow, not checkpoint quality.
Illustrative example: [2,2] and [20,20] both normalize to [0.5,0.5].
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
What information preference does Conv-LLM add to attention?
Convolution for Large Language Models
Show answer and example
Answer: Global matching accesses distant tokens without inherently emphasizing local word combinations. Short convolution first enriches Q/K/V with local order, while attention handles global interaction.
Illustrative example: Aggregate nearby words in “not very good” before reading broader context.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Why add three-tap convolution after Q/K/V despite having attention?
Convolution for Large Language Models
Show answer and example
Answer: Attention selects history by content. Short depthwise residual convolution adds nearby ordered patterns to projected Q/K/V before attention, without replacing global interaction. Left padding preserves causality.
Illustrative example: Local context can combine negation with already seen neighboring words before matching faraway context, but cannot read an unseen word.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How does depthwise QKV convolution differ from replacing attention with convolution?
Convolution for Large Language Models
Show answer and example
Answer: It enriches projections without removing global attention. Depthwise kernels do not fully mix channels, keeping overhead small, and attention still serves long-range access.
Illustrative example: Combine three adjacent positions, then still read a name hundreds of words earlier.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
How should QKV residual convolution handle padding, activation, and bypass?
Convolution for Large Language Models
Show answer and example
Answer: Use causal left padding, add the convolution to projected Q/K/V, then run attention. Do not insert normalization or activation into this documented bypass.
Illustrative example: Width three at t uses t, t−1, t−2, not symmetric future padding.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why make recurrent state queryable rather than only use it as a final summary?
DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling
Show answer and example
Answer: Continuous updates can erase old information. DART retains chunk contributions for targeted access, giving long-range dependencies an addressable path.
Illustrative example: Retrieve chapter one's state to answer a name question after many chapters.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Why keep recurrent states from several chunks instead of one final DART state?
DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling
Show answer and example
Answer: Fixed-capacity recurrence can overwrite old facts. DART decodes chunk-state contributions into queryable K/V, attends to them from the current token, and fuses memory reads with Mamba. Addressability costs extra storage and retrieval.
Illustrative example: A later chapter may overwrite an earlier author's name; retrieve the first chunk instead of relying on the final state.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How does DART state memory differ from storing all token KV?
DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling
Show answer and example
Answer: It reads K/V decoded from compressed recurrent chunks, not original K/V for every token. This trades storage and access flexibility and can still lose detail.
Illustrative example: Retrieving a chunk representation need not reconstruct every word.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Which path should DART's gate g_t modulate?
DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling
Show answer and example
Answer: Mamba is the main output; add gated state-memory attention. Separate recurrent updates, state-to-K/V decoding, and query reads. Do not gate the entire sum.
Illustrative example: Near-zero g_t suppresses memory reads while preserving Mamba.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
How does power-law memory differ from a recent window?
Fractional State Space Transition for Long Sequence Modeling
Show answer and example
Answer: Old influence fades gradually instead of becoming zero at a fixed boundary. Frac-SSM approximates this with finite exponential modes at several timescales.
Illustrative example: A past event can still matter weakly beyond the latest 100 steps.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
How do finitely many recurrent states approximate long fractional-order power-law memory?
Fractional State Space Transition for Long Sequence Modeling
Show answer and example
Answer: One exponential has one decay timescale. Log-spaced modes approximate a power-law tail: fast modes handle recent changes and slow ones keep older influence. Finite capacity and approximation range do not give unlimited exact memory.
Illustrative example: Fast, medium, and slow channels combine into a longer tail than one decay.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
Why do multiple decay rates approximate a long tail better?
Fractional State Space Transition for Long Sequence Modeling
Show answer and example
Answer: Log-spaced modes span recent and distant timescales, unlike a single exponential. Their finite combination approximates a bounded range, not infinite exact history.
Illustrative example: Fast modes track fluctuations while slow ones retain trends.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
What should finite-mode recurrence tests verify besides shape?
Fractional State Space Transition for Long Sequence Modeling
Show answer and example
Answer: Verify equivalence to the finite-mode calculation, long-tail invariants, and state propagation. Operator checks are not reproduction of 1.3B training or model quality.
Illustrative example: Inject an early impulse and compare combined decay over time with the defined approximation.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Does the Gated Attention gate decide where to read or how much to emit?
Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
Show answer and example
Answer: Attention first selects and aggregates positions. The current input then controls how much of each SDPA head output goes forward.
Illustrative example: A head can read a formatting marker but emit less when irrelevant.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Why put the Gated Attention gate after SDPA rather than in logits?
Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
Show answer and example
Answer: Aggregate each head first, then use an input-conditioned sigmoid before output projection. It controls how much of what was read is passed onward and adds nonlinearity between aggregation and projection. This is not changing historical softmax probabilities.
Illustrative example: Suppress a head's formatting output while retaining another head's answer evidence.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
Why isn't a pre-SDPA sigmoid equivalent to a post-SDPA gate?
Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
Show answer and example
Answer: Gating scores or values changes aggregation itself. The post-SDPA gate modulates the already aggregated output; placement is a documented ablation result.
Illustrative example: Scaling one contribution before combining differs from scaling the whole combined result.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Can a fixed constant replace a query-dependent per-head gate?
Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
Show answer and example
Answer: A constant loses input dependence and head selectivity. Produce sigmoid gates from current hidden state and multiply SDPA output before projection. Local gate statistics only verify a nontrivial path.
Illustrative example: Different questions over the same article can use different gates instead of always multiplying by 0.5.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why compress raw byte blocks first?
Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection
Show answer and example
Answer: Compression ratio is a training-free proxy for unique byte density and guides long-range edges, not a semantic importance label.
Illustrative example: Repeated templates compress differently from unique IDs, but hard-to-compress gibberish can be useless.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Why give hard-to-gzip blocks long-range edges, and are they necessarily important?
Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection
Show answer and example
Answer: Blocks above the sample's average compression-ratio threshold are treated as literal-rich and connected at range while preserving local windows and causality. Compressibility is a training-free proxy, not semantic gold; random noise can also be incompressible and too many literal blocks make edges costly.
Illustrative example: An unusual ID may need cross-section access while a repeated template compresses easily, but incompressible gibberish need not help.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How does a gzip mask vary compared with a fixed BigBird or Longformer pattern?
Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection
Show answer and example
Answer: Fixed masks use predetermined positions; gzip uses per-example block compressibility for literal edges and retains local windows. It avoids learned mask weights but pays for compression and construction, so edge counts do not establish speed.
Illustrative example: Moving a unique passage can move gzip edges without changing a positional pattern.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Why intersect literal, local, and self edges with the causal mask?
Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection
Show answer and example
Answer: Their union expresses relevance, not chronology. Intersect it with causality and assign the appropriate masks to the three head types. Masking dense local scores does not accelerate a sparse kernel.
Illustrative example: Selecting two literal blocks cannot let the early one read future tokens in the other.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why combine explicit historical reads and recurrent summaries?
Hymba: A Hybrid-head Architecture for Small Language Models
Show answer and example
Answer: Attention directly retrieves positions, while the SSM compresses history into state. The same layer can use both without forcing either to carry every dependency.
Illustrative example: Explicit access can recover a name while recurrence carries grammar context.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Why put attention and SSM in parallel within Hymba layers?
Hymba: A Hybrid-head Architecture for Small Language Models
Show answer and example
Answer: The same input gets direct historical reads and a recurrent summary in one layer, then fuses both. This combines access mechanisms, not free compute; fusion, KV, and state budgets still matter.
Illustrative example: One branch retrieves an old name while the other maintains running grammar state.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How does parallel Hymba differ from alternating attention and SSM layers?
Hymba: A Hybrid-head Architecture for Small Language Models
Show answer and example
Answer: Parallel branches both read the same layer input and fuse in one update. Alternating layers transform the first branch's result before the second computation. The graphs and fusion times differ at equal layer count.
Illustrative example: Two simultaneous readers share the original, unlike a second reader receiving only the first reader's summary.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Does the local Hymba path include all original features and throughput optimizations?
Hymba: A Hybrid-head Architecture for Small Language Models
Show answer and example
Answer: No. It fuses causal attention and a depthwise convolutional state branch, without meta tokens, 1.5B scale, or fused kernels. A runnable branch does not establish paper throughput.
Illustrative example: Passing a small-model gated forward is a mechanism check, not a deployment speed measurement.
Sections in the Chinese source: 复现边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
What memory quantity does KronLSTM correct?
Kronecker Delta Rule for Long Context Linear Attention
Show answer and example
Answer: It corrects scalar association vᵀSk rather than forcing the entire retrieved vector to equal v. The update follows vkᵀ with separate forget and erase/write gates.
Illustrative example: Adjusting one association need not rewrite every value direction for that key.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Does KronLSTM update a full value vector or one scalar key–value association?
Kronecker Delta Rule for Long Context Linear Attention
Show answer and example
Answer: It uses vᵀSk and updates along vkᵀ, unlike correcting an entire vector error. Interference depends on overlap in both key and value; orthogonality on either side blocks that update's interference. Forget, erase, and write are separate; do not multiply the write coefficient by the erase gate.
Illustrative example: Similar keys with orthogonal values do not interfere under one scalar association update, without guaranteeing no future interference.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How does KronLSTM's error differ from DeltaNet for interference analysis?
Kronecker Delta Rule for Long Context Linear Attention
Show answer and example
Answer: DeltaNet uses Sk−v, while KronLSTM uses scalar vᵀSk−c. Interference involves both key and value inner products; one orthogonal side removes interference from this write, not all future forgetting.
Illustrative example: Similar keys with orthogonal values can avoid interference despite key overlap.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Why not copy ordinary delta update's forget direction and write coefficient?
Kronecker Delta Rule for Long Context Linear Attention
Show answer and example
Answer: Forgetting acts on the right along the key axis. The decoupled write is c·vkᵀ, not βc·vkᵀ; only erase uses β. Compare an independent vec reference instead of checking loss alone.
Illustrative example: Small β should not suppress the independent write gate c.
Sections in the Chinese source: 核心公式, 实现与运行. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why does Mamba selectivity depend on the current token?
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Show answer and example
Answer: Step size and read/write vectors depend on input, so retention, updates, and readout vary by content while still using recurrent state.
Illustrative example: A separator and answer token can trigger different updates.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
What does Mamba's selectivity change inside the SSM?
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Show answer and example
Answer: Token-dependent step sizes and read/write vectors control retention, write direction, and readout within recurrence. This is not output scaling of a fixed SSM. Linear sequence cost still does not guarantee exact arbitrary-token retrieval from finite state.
Illustrative example: Preserve state on filler but write differently for a key number; retrieval still depends on what remains.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How does Mamba memory differ from reweighting historical KV?
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Show answer and example
Answer: Mamba accumulates history in state and uses input-dependent readout. Attention assigns weights directly to retained positions. Linear complexity does not imply lossless per-token history.
Illustrative example: A summary can retain a fact without separately addressable slots for every old word.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Does a Python or PyTorch selective scan reproduce Mamba kernel throughput?
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Show answer and example
Answer: No. A loop can check input-dependent parameters and recurrence, but not a fused parallel CUDA scan. The local run also omits 2.8B pretraining. Validate correctness, language quality, and throughput separately.
Illustrative example: Seven steps agreeing with a reference check mechanics, not tokens per second.
Sections in the Chinese source: 本地复现, 复现边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why evaluate test-time learning rather than IID performance alone?
Open-world AI: a Tale of Three Requirements
Show answer and example
Answer: IID testing does not show use of newly observed task experience. Mosaics+ combines recent dependence and long-term predictive associations to organize new-task evidence.
Illustrative example: Use newly observed input/output examples to infer a rule rather than repeat old statistics.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Why normalize current and related-task memory separately in Mosaics+?
Open-world AI: a Tale of Three Requirements
Show answer and example
Answer: One softmax lets numerous irrelevant past entries dilute current evidence. Two-stage retrieval normalizes each source then combines predictions. Short-term memory handles recent dependence, long-term stores predictive associations, and illegal future reads stay excluded.
Illustrative example: Three relevant current entries should not be diluted solely because there are 1,000 old records.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How does two-stage Mosaics+ differ from a single softmax pool?
Open-world AI: a Tale of Three Requirements
Show answer and example
Answer: Separate current and related-task normalization precedes combining. In a shared pool, adding even unrelated old entries can change current weight; these are different computations.
Illustrative example: Many old items should not drown out a few important current associations by count alone.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
How does predictive value at i+1 avoid a future read?
Open-world AI: a Tale of Three Requirements
Show answer and example
Answer: Restrict i≤t−s with s≥1. Then i+1 is within the visible boundary. Split normalization at document boundaries and handle empty memory; reference loops and future perturbations are negative controls.
Illustrative example: With skip=1, i=t−1 may use t but not t+1.
Sections in the Chinese source: 核心公式, 结果与独立对照. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Is an MSA history block a new layer or a token-axis candidate?
Show answer and example
Answer: Blocks organize history for indexing. A light branch chooses blocks and the main branch reads their tokens. Block selection and exact attention have separate costs.
Illustrative example: Choosing a chapter in a table of contents differs from reading its sentences.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
What does MSA save and preserve by selecting blocks before exact attention?
Show answer and example
Answer: A cheap block index reduces the historical range; main attention computes exact token interactions inside hits. Summaries select rather than replace tokens. A missed answer block cannot be recovered by exact attention on the rest.
Illustrative example: Choose two of ten pages to read word by word, saving main attention on eight but missing a date if its only page was dropped.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
Does MSA indexing do the same thing as shrinking main attention KV?
Show answer and example
Answer: The index decides where to read; main attention keeps its own exact Q/K/V inside selected blocks. Compressing KV changes what representations are read, not only where.
Illustrative example: Selecting two original passages preserves different evidence from summarizing the whole text into two passages.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Why doesn't reduced MSA attention pair ratio imply the same speedup?
Show answer and example
Answer: Runtime includes indexing, selection, movement, and kernels. The unfused PyTorch/MPS path has fewer pairs but takes longer. Report pair ratio, PPL, and measured time separately.
Illustrative example: An 80% pair reduction can still slow training if overhead dominates.
Sections in the Chinese source: 本地复现. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why group history into blocks instead of reading all of it for every query?
MoBA: Mixture of Block Attention for Long-Context LLMs
Show answer and example
Answer: Routing picks a few relevant blocks, then exact attention reads inside them. This reduces candidate pairs while keeping within-block detail.
Illustrative example: A question needing two passages need not attend to every word.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Why keep the current causal block when MoBA routes to a few remote ones?
MoBA: Mixture of Block Attention for Long-Context LLMs
Show answer and example
Answer: Routing finds distant evidence; current-block retention protects nearby context from routing misses. Causal masking restricts visible positions, and attention inside selected blocks is exact at token level. Sparsity does not preserve every distribution.
Illustrative example: Keep the current subject even if remote chapters score higher; do not read the current block's ungenerated suffix.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
What can content-routed MoBA recover that a recent window cannot?
MoBA: Mixture of Block Attention for Long-Context LLMs
Show answer and example
Answer: A window excludes positions by age. Routing can recover relevant distant blocks but can still miss evidence. Cost depends on block size and routing budget; it is not lossless pruning.
Illustrative example: A name at the beginning can return through content matching instead of being automatically excluded.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Does selecting the current MoBA block remove the need for causality inside it?
MoBA: Mixture of Block Attention for Long-Context LLMs
Show answer and example
Answer: No. Mask positions after the query inside the selected block. The local eight-token, top-two configuration is not a million-context fused kernel.
Illustrative example: Word three cannot read word seven merely because both share an eight-token block.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
What retrieval instability is Möbius RoPE intended to address?
Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable
Show answer and example
Answer: Standard RoPE long-range needle retrieval can vary strongly by training seed. Antiperiodic frequencies on some heads aim to stabilize cross-window phase while leaving others unchanged.
Illustrative example: Identical needles in long text can produce very different retrieval results across seeds.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
How does Möbius position encoding differ from standard RoPE across a training window?
Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable
Show answer and example
Answer: Antiperiodic frequencies on some heads yield −I at one window and I at two. Other heads retain standard RoPE for language modeling. This changes positional geometry, not accessible history or guaranteed retrieval at all lengths.
Illustrative example: At N positions the selected heads reverse phase; at 2N they realign. Actual retrieval still depends on Q/K and training.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
What explicit frequency boundary does Möbius add?
Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable
Show answer and example
Answer: Set some two-dimensional frequencies to π(2i+1)/N, producing −I across N and I across 2N. Other heads keep standard RoPE. This is geometry, not measured improvement.
Illustrative example: Reverse phase after one window and recover after two, not arbitrary scaling across every head.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Should Möbius RoPE change N dynamically with inference length?
Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable
Show answer and example
Answer: No. N is fixed training context; local N=64 replaces one of four heads. Dynamic N changes frequencies for identical positions and breaks the boundary definition. Retain negative local results.
Illustrative example: Length-32 and length-80 inputs both use the same training N.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why give MS-GLA heads different temporal scales?
Show answer and example
Answer: One resolution can bottleneck representations. Some heads preserve token-level variation, others use pooled history, and their outputs combine.
Illustrative example: Phrase detail and trends across phrases can use different update rates.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
How does MS-GLA avoid leaking later tokens when pooling a block?
Show answer and example
Answer: Each coarse branch masked-averages nonoverlapping blocks and runs GLA. A summary becomes visible only at the block end; causal hold restores token resolution without broadcasting into earlier positions. Input-conditioned weights fuse scales.
Illustrative example: In a four-token block, position two cannot see the average including positions three and four.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
Why is multiscale GLA more than a larger window?
Show answer and example
Answer: Each scale pools at a different nonoverlapping granularity before GLA, changing temporal update resolution. Extending a window still updates at the original rate.
Illustrative example: Updating once per four words differs from updating every word while looking farther back.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
When may tokens access scale-four pooled state?
Show answer and example
Answer: At or after the block's final token. Use causal hold and input softmax fusion rather than broadcasting full means to every token.
Illustrative example: Token one cannot use the average of words one through four before word four arrives.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why independently control old-state retention and new writes?
Show answer and example
Answer: A shared complementary gate creates a tradeoff. Separate retain/write gates in the discrete transition let both adjust independently.
Illustrative example: Reading an important new fact need not erase an old fact.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Why separate retention and writing rather than use complementary gates in Naju?
Show answer and example
Answer: A complementary write would shrink whenever retention is strong. Independent gates on native discrete dynamics can preserve old state and write new evidence together. Token-dependent directions, convolution, and feedthrough still matter.
Illustrative example: Keep a name while adding age instead of erasing the name in proportion to the new write.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How do Naju's native discrete poles differ from Mamba discretizing continuous dynamics?
Show answer and example
Answer: Naju directly parameterizes discrete retention. Mamba derives a transition from continuous dynamics and input-dependent time steps. Both recur but formulas and gate coupling differ.
Illustrative example: One learns the retained fraction per step directly; the other derives it from a time step and continuous matrix.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
How should preserve-first initialization set Naju's two gates?
Show answer and example
Answer: forget_bias=5 and write_bias=−2 create strong initial retention and weak writes. Retain token-dependent B/C and direct feedthrough.
Illustrative example: Sigmoid(5)≈0.993 and sigmoid(−2)≈0.119 are not equal gates.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why use compression, selection, and window paths together?
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Show answer and example
Answer: They provide global overview, exact selected detail, and exact recent context, respectively. Learned gates combine them; one sparsity pattern need not serve all roles.
Illustrative example: Get topic summary, a distant key sentence, and current phrase detail from different paths.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Why retain original blocks and a sliding window in addition to compressed NSA history?
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Show answer and example
Answer: Compression gives cheap global context but can lose details. Query-selected original blocks recover precise evidence, and a window covers recent tokens. All three paths learn and fuse from pretraining and remain causal.
Illustrative example: A summary can identify the chapter about dates; exact selected tokens resolve the year and the window preserves current qualifiers.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How does NSA differ from pruning low attention weights after training?
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Show answer and example
Answer: Three paths and their gates are trained jointly from pretraining, changing what the model learns to access. Post-hoc deletion from a dense model does not provide the same adaptation.
Illustrative example: The model learns to trade summary and detailed blocks rather than suddenly losing history.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Do all three NSA branches need causal boundaries?
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Show answer and example
Answer: Yes. Compressed blocks, selected fine blocks, and windows cannot expose future information. Align output dimensions before gated fusion. Local edge counts do not measure paper Triton throughput.
Illustrative example: Mask future words in a selected block spanning the query position.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why focus on high-contribution post-RoPE QK coordinates?
Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels
Show answer and example
Answer: Coordinate contributions are uneven; a high-contribution tail can dominate error. Grouping and positive quadratic kernels target these differences rather than treating every coordinate alike.
Illustrative example: A few coordinates driving most score changes merit different approximation from low-contribution ones.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Why group QK contribution dimensions and calibrate OAL against output error?
Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels
Show answer and example
Answer: A few coordinates can dominate softmax approximation error. OAL finds stable post-RoPE groups and learns positive-definite quadratic kernels for shared groups and pairs. Vectors after aggregation, not only weights, reveal impact; positivity protects normalization.
Illustrative example: Equal weight errors have different effects if one location has a large or very different value.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How does OAL output MSE differ from fitting attention weights?
Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels
Show answer and example
Answer: Output depends on values and normalized aggregation, so equal weight errors can affect outputs differently. Calibrate the kernel to actual readouts.
Illustrative example: Swapping weights on identical values changes nothing, unlike swapping opposite values.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Which OAL grouping, calibration, and initialization choices may use test data?
Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels
Show answer and example
Answer: None. Train supports grouping, calibration, and LoRA; validation selects initialization and checkpoint; test comes last. Keep ε in the positive kernel to ensure a positive denominator.
Illustrative example: Do not replace the validation-selected kernel with one found better on test and report that same test again.
Sections in the Chinese source: 核心公式, 数据协议. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Are ODA local and remote history divided by importance?
On-Demand Attention: Efficient Long-Context Decoding with Learned Recall
Show answer and example
Answer: No. They describe distance relative to the current token. Relevant evidence can be far and nearby words irrelevant, motivating optional global reads.
Illustrative example: A name at the document start can be critical outside the current window.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
When does On-Demand Attention trigger global retrieval?
On-Demand Attention: Efficient Long-Context Decoding with Learned Recall
Show answer and example
Answer: A cheap recall head predicts benefit before expensive global attention and triggers it only for worthwhile tokens. Full KV stays stored; it saves some reads, not cache capacity. Misses can hide relevant distant evidence.
Illustrative example: Punctuation may use local context, while quoting an opening number triggers global reading; decide before paying for it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How does on-demand retrieval differ from evicting old KV?
On-Demand Attention: Efficient Long-Context Decoding with Learned Recall
Show answer and example
Answer: ODA retains all KV and varies when it is read. Eviction removes historical entries from the accessible set and changes memory capacity as well.
Illustrative example: A token skipped now may still be retrieved next step if retained.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Why must ODA decide before computing global attention?
On-Demand Attention: Efficient Long-Context Decoding with Learned Recall
Show answer and example
Answer: Computing global attention first already pays the cost. The gate must control conditional execution. A local A100 receipt verifies the path, not end-to-end vLLM throughput.
Illustrative example: Reading the whole book before choosing the contents page saves no reading.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why use a controlled OLMpool model pool?
Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension
Show answer and example
Answer: Real models often differ in normalization, GQA, training length, and sliding windows simultaneously. The pool isolates these choices and their combinations for long-context extension.
Illustrative example: If architecture and training length both change, do not attribute the result to one toggle.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Why isn't matching parameter count sufficient for comparing long-context methods?
Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension
Show answer and example
Answer: Normalization, GQA, pretraining windows, and sliding attention can all change long-context behavior. A controlled model pool isolates them; one model's success does not generalize a positional or attention modification.
Illustrative example: A longer-pretrained model beating a shorter-trained model does not isolate the extension method.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How does changing normalization within the pool differ from comparing released models?
Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension
Show answer and example
Answer: Released models can also differ in data, scale, and schedule. Controlled changes improve attribution, and interactions mean single-factor effects need not add.
Illustrative example: Swap normalization alone, then test whether the direction persists with GQA.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Does a passing local mini-suite establish the 7B pool's conclusions?
Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension
Show answer and example
Answer: No. It verifies core data flow, not paper-scale checkpoints or full benchmarks. Causal attribution needs controlled training, matched extension protocols, and task evaluation.
Illustrative example: Accuracy on 64 diagnostic examples is not a controlled 7B long-context experiment.
Sections in the Chinese source: 本地复现, 复现边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why not read only a few important KV entries?
OpenAttention: One Percent Is Enough with Tail Proxies
Show answer and example
Answer: A low read rate can leave substantial total mass in the tail. Counted residual-cluster averages supplement exact reads instead of treating the tail as zero.
Illustrative example: Many individually weak tokens can jointly affect the output.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Why use a count-weighted proxy for unread KV in OpenAttention?
OpenAttention: One Percent Is Enough with Tail Proxies
Show answer and example
Answer: Top-k discards all unread softmax mass. Query-relevant geometry groups tokens, with more exact reads where estimated error is high. Residual mean keys/values enter joint normalization with log counts. This preserves approximate mass, not exact attention.
Illustrative example: A mean proxy for 100 similar tail tokens needs count correction or it represents just one entry.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
What does a residual proxy retain beyond ordinary top-k?
OpenAttention: One Percent Is Enough with Tail Proxies
Show answer and example
Answer: The residual mean and count approximate omitted contributions and share softmax with exact tokens. The mean is not exact recovery.
Illustrative example: A proxy for 100 remaining tokens should not participate with single-token mass.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Why recompute cluster residuals after selecting exact tokens?
OpenAttention: One Percent Is Enough with Tail Proxies
Show answer and example
Answer: Otherwise exact rows are counted twice. Remove them, recompute means and counts, add log count to logits, and omit empty clusters. Keep documented deterministic boundaries rather than heuristic fixes.
Illustrative example: Selecting three of ten leaves a seven-item proxy.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why separate base and output-specific tokens?
Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers
Show answer and example
Answer: Most prefill positions build cache rather than emit outputs. Base uses a fixed window and retains full KV; output-specific queries read it when predictions are needed.
Illustrative example: Most prompt positions in a long document need not perform global output reads.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
How can Orthros decode from all history after local prefill?
Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers
Show answer and example
Answer: Base tokens use sliding attention but build full per-layer KV. Output-specific queries read shared full history where predictions are required and do not write KV. Fixed-window prefill compute is linear, not linear-attention state or reduced cache.
Illustrative example: During prefill base tokens read neighbors, while the final output query and future decode positions can retrieve full stored history.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
Is Orthros linear prefill kernelized linear attention?
Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers
Show answer and example
Answer: No. Fixed-window SWA costs O(Tw). The output stream still performs ordinary causal cross-attention over full KV; it neither compresses into recurrence nor discards old cache.
Illustrative example: KV at the start remains available even if most prompt positions never read it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Should Orthros's output stream append its own K/V?
Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers
Show answer and example
Answer: No. Same-layer KV comes only from base. Output queries use a shared projection and stream-specific affine transforms. Writing output KV changes the contract and may break cached-versus-batch parity.
Illustrative example: Append one base KV per step, not one per stream.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why not top-k system instructions with chat history?
Role-aware Heuristic Episodic Attention for Conversational LLMs
Show answer and example
Answer: Global instructions persist across turns. Episodic content is retrieved separately and handled by role.
Illustrative example: A lasting formatting requirement should not be evicted for low similarity to the current query.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Why keep global instructions in a persistent REA prefix?
Role-aware Heuristic Episodic Attention for Conversational LLMs
Show answer and example
Answer: Relevance ranking should not discard standing constraints. REA separates them from episodic retrieval, which applies role-specific preservation, compression, or omission. Context selection is not lossless reconstruction; policies require verification.
Illustrative example: A standing Chinese-output requirement must not disappear during a movie discussion merely for low keyword similarity.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How does role-aware episodic retrieval differ from a recent window?
Role-aware Heuristic Episodic Attention for Conversational LLMs
Show answer and example
Answer: It can retrieve distant relevant interactions instead of using recency alone, and retains persistent instructions independently. Role policies treat content differently; compression may lose information.
Illustrative example: Recover an older preference while dropping recent irrelevant tool logs.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Does the local REA top-k path include real LLM compression?
Role-aware Heuristic Episodic Attention for Conversational LLMs
Show answer and example
Answer: No. It executes role separation and episodic retrieval without a compression LLM. Distinguish selection from generated summaries.
Illustrative example: Choosing three passages does not show an LM compressed them faithfully.
Sections in the Chinese source: 复现边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why rotate two-dimensional Q/K subspaces?
RoFormer: Enhanced Transformer with Rotary Position Embedding
Show answer and example
Answer: Position determines rotation angle, so dot products depend on relative phase and content without learning one vector per position.
Illustrative example: Identical content at distances 2 and 20 can produce different scores.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Why do absolute Q/K rotations yield relative position in RoPE dot products?
RoFormer: Enhanced Transformer with Rotary Position Embedding
Show answer and example
Answer: The first rotation's transpose cancels the shared absolute phase, leaving displacement n−m. Q/K subspaces rotate, not values. Evaluability at larger positions does not guarantee trained retrieval extrapolation.
Illustrative example: Moving positions 3 and 8 to 13 and 18 preserves distance 5 and their relative phase.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How does RoPE differ from adding absolute embeddings?
RoFormer: Enhanced Transformer with Rotary Position Embedding
Show answer and example
Answer: Absolute embeddings modify representations additively. RoPE rotates Q/K so the inner product can express a relative rotation. These are different operations.
Illustrative example: Shifting both segments equally preserves RoPE's relative phase.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Can RoPE rotate only Q or only K?
RoFormer: Enhanced Transformer with Rotary Position Embedding
Show answer and example
Answer: No. Relative displacement requires both R_m and R_n. Pair channels correctly and continue absolute positions through incremental decode.
Illustrative example: A single input token at position 100 still uses rotation 100, not zero.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why keep continuous SAS gates rather than only selected indices?
SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
Show answer and example
Answer: The integer indices choose blocks but do not carry gradients. Gates in attention let prediction loss train selection scores.
Illustrative example: Identical selected blocks with 0.8/0.2 versus 0.5/0.5 produce different outputs and gradients.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
How does SAS train its selector despite nondifferentiable top-k indices?
SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
Show answer and example
Answer: It preserves continuous gates for selected blocks and adds log gates inside attention softmax. LM loss differentiates through gate values, not indices. History gates normalize jointly and the current block stays at one.
Illustrative example: Changing two selected gates from 0.8/0.2 to 0.6/0.4 changes output and gradients without changing indices.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How does LM-loss selector training differ from distilling dense attention rankings?
SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
Show answer and example
Answer: Token prediction provides task feedback rather than supervising imitation of dense attention. It needs continuous gates; hard top-k remains nondifferentiable.
Illustrative example: A block ranked highly by the dense model need not contribute equally to prediction loss.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Where does SAS add the log gate?
SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
Show answer and example
Answer: Add log gates for selected history to attention logits before shared softmax. Normalize history gates together and set current block to one. Post-output scaling changes the normalization relationship.
Illustrative example: Current gate one adds zero; history blocks compete with their respective log gates.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why retain heavy entries deterministically and sample the tail?
Stratified Sparse Attention: Reducing the Sparse-to-Dense Gap at Extreme Sparsity
Show answer and example
Answer: Top-k drops the tail; uniform sampling can waste budget. SSA uses proxy scores to divide deterministic and residual budgets, then corrects by inclusion probability.
Illustrative example: Strong evidence remains while weaker entries have a chance of inclusion.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Why correct by the probability of at least one draw after deduplicating SSA samples?
Stratified Sparse Attention: Reducing the Sparse-to-Dense Gap at Extreme Sparsity
Show answer and example
Answer: A repeated token appears only once in the set, so inclusion is 1−(1−p)^m rather than single-draw p. Deterministically retained heavy items have inclusion one. The final self-normalized ratio is not strictly unbiased.
Illustrative example: At p=0.2 and two draws, weight one retained row using 0.36, not 0.2.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
Why not correct deduplicated with-replacement samples by 1/p_i?
Stratified Sparse Attention: Reducing the Sparse-to-Dense Gap at Extreme Sparsity
Show answer and example
Answer: Each token enters once with π_i=1−(1−p_i)^m. Divide by inclusion, not per-draw probability. The self-normalized numerator/denominator ratio still need not be unbiased.
Illustrative example: A token drawn three times but retained once is not three independent weighted observations.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
How does SSA compute small inclusion probabilities and test full budget?
Stratified Sparse Attention: Reducing the Sparse-to-Dense Gap at Extreme Sparsity
Show answer and example
Answer: Use −expm1(m·log1p(−p)) and log-space normalization for stability. Return dense attention explicitly at k=N. Detached discrete allocation is not a gradient path.
Illustrative example: A budget of all keys must match dense output without randomly losing tail tokens.
Sections in the Chinese source: 核心公式, 验证与边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
What can Switch Attention's local and full paths read?
Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformers
Show answer and example
Answer: Local reads the visible window; full reads all causal history, never the future. The router trades evidence range and compute per token.
Illustrative example: A nearby phrase may need only local context, while an old fact needs a wider read.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Does Switch Attention choose one path per layer or per token?
Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformers
Show answer and example
Answer: A fine-grained router chooses full or local for each token. Training must account for discrete selection; actual savings depend on global call rate and overhead, and sparsity does not guarantee quality.
Illustrative example: Nearby conjunctions can use local reads, while citing an early name may need global history in the same layer.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How does hard switching differ from continuously mixing both branches?
Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformers
Show answer and example
Answer: Hard selection can execute only one branch. Continuous mixing generally computes both and weights outputs. Gradients and savings differ; the local soft router is not the paper's STE hard router.
Illustrative example: Multiplying an already computed attention output by nearly zero does not save its execution.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Can the local soft router demonstrate a branch-selective kernel speedup?
Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformers
Show answer and example
Answer: No. It checks mixing without hard routing, 32K continued pretraining, or fused selective kernels. Verify actual executed branches and runtime separately from gate values.
Illustrative example: A 90% local weight does not prove 90% of global calls were skipped.
Sections in the Chinese source: 复现边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Does a frozen TransMem backbone mean nothing learns?
TransMem: Transforming Hidden States into Memory for Large Language Models
Show answer and example
Answer: No. The backbone stays fixed but evidence-conditioned self-distillation trains memory gating. Changing memory is not updating backbone weights.
Illustrative example: A new branch can learn when to inject history into the current state.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
How does parameter memory from hidden states differ from retaining original history?
TransMem: Transforming Hidden States into Memory for Large Language Models
Show answer and example
Answer: TransMem converts sparse states from a frozen backbone to reusable gated deltas. It avoids re-reading the same original tokens but trades reuse for detail lost in selection/compression. Learned gating does not retrain the backbone per history.
Illustrative example: A compressed document can support later queries without recovering an omitted exact ID.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How does parameter memory differ from appending old text?
TransMem: Transforming Hidden States into Memory for Large Language Models
Show answer and example
Answer: T(H_sparse) transforms selected states into hidden-state updates, while text concatenation re-reads original tokens. This changes storage and fusion, and omitted detail may be lost.
Illustrative example: Keeping a fact representation does not reproduce every omitted word.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Should TransMem replace the original hidden state?
TransMem: Transforming Hidden States into Memory for Large Language Models
Show answer and example
Answer: No. h′=h+g⊙T(H_sparse) preserves the main path. Keep selection, transformation, and gated injection distinct. The mini-suite is not paper-scale training.
Illustrative example: With g=0, output is h rather than zero.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Is linear-attention state a history list or accumulated associative memory?
Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling
Show answer and example
Answer: It accumulates key/value outer products into a matrix queried by contraction. Triadic adds another key axis, not a raw per-token KV list.
Illustrative example: Multiple writes combine in a table with an added address dimension.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
How does a second key change Triadic Linear Attention?
Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling
Show answer and example
Answer: It adds an independently addressed key axis to ordinary key–value state and contracts with two queries. Capacity grows with that dimension, not for free. E=1 should recover ordinary linear attention.
Illustrative example: A person's value can depend on the scene as a second address, at the cost of a larger joint state.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How is a second key axis different from widening values?
Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling
Show answer and example
Answer: Dimension E increases state capacity by E and enables an independent query contraction. Wider values add output content but not dual-key addressing.
Illustrative example: Adding more fields to records is different from adding a search key.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Why test the E=1 limit instead of merely checking forward execution?
Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling
Show answer and example
Answer: It must exactly reduce to ordinary linear attention, testing construction and contraction of the extra axis. Also compare explicit history sums, causality, and gradients; this is not long-book language quality.
Illustrative example: Remove the second key's freedom and match the same-input two-dimensional reference.
Sections in the Chinese source: 背景与主要改动, 本地复现. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
What information differs between teacher and student?
Learning What to Remember: Test-Time Training via Context Distillation
Show answer and example
Answer: Teacher sees longer history, while student relies on its window or fast weights. Hidden-state targets teach retention without giving deployed students full history.
Illustrative example: A teacher reads background; deployment still uses finite learned memory.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
What does a short-window TTCD student learn beyond final answers?
Learning What to Remember: Test-Time Training via Context Distillation
Show answer and example
Answer: Long-window teacher hidden states supervise fast weights to retain context useful for future predictions. The student must select information under finite memory, not uniformly preserve every token. State distillation is not lossless.
Illustrative example: Store an early variable definition in fast weights to match later teacher state, rather than memorizing one answer.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
What does state distillation constrain beyond final token matching?
Learning What to Remember: Test-Time Training via Context Distillation
Show answer and example
Answer: It aligns intermediate teacher and student representations to train memory, not just output probabilities. Supervision location and space differ.
Illustrative example: Two students can predict the same token while retaining different historical information.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Why align teacher and student positions and representations?
Learning What to Remember: Test-Time Training via Context Distillation
Show answer and example
Answer: Hidden loss compares the same t with compatible dimensions. Offsets or accidental broadcasting optimize the wrong target. Local results are not paper-scale long-context validation.
Illustrative example: Student step 20 aligns to teacher step 20, not the final document state.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Does test-time compression retrain model parameters at inference?
Rethinking Sequence Compression with Test-time Regression
Show answer and example
Answer: It solves coefficients anew for each token group. Outer loss learns target projection and regularizer; closed-form per-input fitting does not update those parameters at inference.
Illustrative example: Two inputs share a and ρ but solve different h.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Why normalize TTR keys but preserve target magnitudes?
Rethinking Sequence Compression with Test-time Regression
Show answer and example
Answer: Unit keys group by direction; targets retain token strength. A static projection from the same normalized keys could be fit trivially and lose input-dependent compression. Per-forward coefficients do not update persistent parameters.
Illustrative example: Same-direction tokens of norm one and three have the same key but different targets; normalizing both would erase that distinction.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
How does TTR differ from fixed mean pooling?
Rethinking Sequence Compression with Test-time Regression
Show answer and example
Answer: It formulates and exactly solves a group-dependent ridge regression. Differentiating through the solve trains outer parameters; this is a different compression objective, not just new weights.
Illustrative example: Groups with equal means but different directional correlations can yield different coefficients.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Why solve a T×T token-space system, and why not detach?
Rethinking Sequence Compression with Test-time Regression
Show answer and example
Answer: Scale K by D inverse square root, solve I_T plus the scaled K Gram matrix by Cholesky, then map to d dimensions. The outer loss needs solve gradients for projection and regularization. Complexity retains a cubic T term.
Illustrative example: Token-space is attractive when T≪d but is not always only O(T²d).
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Does Select-TTT change retrieval or the strength of writes to fast weights?
Select-TTT: Query-Conditioned Selective Fast Weight Updates for In-Place Test-Time Training
Show answer and example
Answer: It keeps In-Place TTT chunk writes and scales each chunk's learning rate with question relevance. Scores come from question-token negative log probabilities under the frozen LM head. It neither trains a retriever nor discards all low-scoring chunks. Different questions can induce different multipliers.
Illustrative example: Example: A long conversation may favor different passages for a moving date and food preferences, without labels telling the model where the answer is.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Why average the lowest-CE positions in a chunk rather than the highest?
Select-TTT: Query-Conditioned Selective Fast Weight Updates for In-Place Test-Time Training
Show answer and example
Answer: CE is mean negative log probability of question tokens under the current hidden state's vocabulary distribution. Lower values mean the position predicts question words better. Averaging the lowest k preserves local relevance more robustly than a single minimum. Reversing it reinforces irrelevant positions.
Illustrative example: Example: For CE 1 and 6, select 1 as more relevant. Repeated question tokens count as many times as they appear.
Sections in the Chinese source: 核心公式, 检查与负控. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
Why isn't Select-TTT a strictly conserved write-budget allocation?
Select-TTT: Query-Conditioned Selective Fast Weight Updates for In-Place Test-Time Training
Show answer and example
Answer: Softmax multiplied by chunk count sums to that count before clipping. An upper cap can lower the sum. Approximate mean learning-rate preservation is not strict conservation. Renormalization changes Eq. 10 and can violate the per-chunk cap.
Illustrative example: Example: Three chunks can start near [3,0,0]. A cap of 1.2 gives total about 1.2, not 3; this is a formula example, not a QA result.
Sections in the Chinese source: 核心公式, 检查与负控. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Why emit the current chunk before writing its keys and values?
Select-TTT: Query-Conditioned Selective Fast Weight Updates for In-Place Test-Time Training
Show answer and example
Answer: Its outputs read the pre-update state; only later chunks use the new information. Writing first leaks the current update into an earlier output and changes causal order. The local reference tests tensor-state transitions, not real NTP-aligned features, backbone integration or long-dialogue evaluation.
Illustrative example: Example: Start with W=0. Chunk one outputs 0 then writes W=1. Chunk two can read 1; an output of 1 in chunk one would use its write prematurely.
Sections in the Chinese source: 核心公式, 本地复现, 检查与负控. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Is Sleep just a pause or a separate learning stage?
Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
Show answer and example
Answer: It is offline consolidation. Before applying precomputed base updates, expand slow modules and use the pre-update teacher to consolidate fast memories into the student to be updated. Then apply the update and clear sending-layer experts. It is not just a pause or brief replay, and the full chain is not implemented locally.
Illustrative example: Example: Verify notes copied into a long-term library before deleting the temporary notes. Computing note similarity is not consolidation.
Sections in the Chinese source: 背景与主要改动, 高成本完整实验. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Does LTI's edit-distance threshold cap edit count or a similarity ratio?
Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
Show answer and example
Answer: It caps the absolute number of token edits, inclusively. For accepted pairs the literal reward is one minus distance divided by the longer completion length. A fixed similarity-ratio threshold differs.
Illustrative example: Example: Threshold 1 admits one substitution in both 6- and 60-token completions, with rewards 5/6 and 59/60.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
Does crossing the literal-distance threshold also zero semantic reward?
Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
Show answer and example
Answer: No. A separate frozen judge assigns binary semantic reward. The threshold only gates the literal branch. Very different wording can preserve meaning; replacing the judge with edit distance changes the method.
Illustrative example: Example: With gamma 0.4 and a positive semantic judgment, a zero literal branch still yields total reward 0.4.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Can the local Sleep reward prove the model retains knowledge long term?
Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
Show answer and example
Answer: No. It receives tokenized completions and external semantic labels and checks reward boundaries. No dream sampling, CMS, expansion, LoRA, screening or forgetting evaluation. The costly full experiment is not reproduced; the kernel is not an Evolve capability operator.
Illustrative example: Example: Reward 0.9 for two integer sequences confirms the formula for that input, not any update to model weights.
Sections in the Chinese source: 本地复现与边界, 高成本完整实验. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Basics
Why can two key segments with the same mean require different attention routing?
Multi-State Linear Attention with Generalized Second-Order Routing
Show answer and example
Answer: Mean alignment can hide strong key matches canceled by others. MERLIN also retains raw second moments and recovers query-direction alignment variance, mixing it with mean routing. It enriches segment routing without reconstructing full per-token attention.
Illustrative example: Example: Keys [-2,2] and [0,0] both average zero. Query 1 gives alignment variances 4 and 0, invisible to mean routing.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Core mechanism
Why does Eq. 7 distinguish the second moment from covariance?
Multi-State Linear Attention with Generalized Second-Order Routing
Show answer and example
Answer: It stores mean(kkᵀ), without centering, then computes variance qᵀΣq−(qᵀμ)². Passing covariance and subtracting again double-centers; treating the raw moment as variance includes mean strength. The local reference directly measures dot-product variance.
Illustrative example: Example: Keys [1,3] have mean 2, moment 5 and variance 1. Using 5 overcounts by 4; subtracting 4 from centered 1 yields the invalid −3.
Sections in the Chinese source: 核心公式, 检查与负控. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Method comparison
Why not mix logits then apply one softmax instead of mixing two softmax distributions?
Multi-State Linear Attention with Generalized Second-Order Routing
Show answer and example
Answer: Softmax is nonlinear. MERLIN normalizes mean and variance routes separately and mixes probabilities with gamma. Mixing logits first changes competition and normalization even though both outputs sum to one.
Illustrative example: Example: Mean logits [4,0], variance logits [0,0], gamma 0.5 yield [0.741,0.259]; softmax of mixed logits [2,0] gives [0.881,0.119].
Sections in the Chinese source: 核心公式, 检查与负控. The example illustrates the mechanism.
Foundation models / Long context: attention, recurrent state, and memory
Implementation details
Why only expose completed segments to local queries?
Multi-State Linear Attention with Generalized Second-Order Routing
Show answer and example
Answer: Full-segment means and moments include future tokens for queries inside that segment. A full model could maintain prefix moments and recurrent state; the local path does not. It exposes completed segments only and rejects calls with none visible, rather than leaking future statistics.
Illustrative example: Example: Segment start 4 length 4 covers positions 4–7. It is not visible at 6, only at 7; reading all keys at 4 leaks positions 5–7.
Sections in the Chinese source: 本地复现, 检查与负控. The example illustrates the mechanism.
Topic guide and reading order
Start with this question¶
As context grows, does the model preserve addressable historical tokens or a compressed recurrent state?
Takeaway¶
Separate position, information access, and state capacity. RoPE rotates two-dimensional Q/K subspaces by position so dot products capture relative offsets; attention still accesses history within its window. Mamba makes state-update steps and read/write parameters input-dependent and compresses history into recurrent state. Sparse and hybrid attention lie between these approaches, reading selected history or retaining full access in some layers. MERLIN aggregates reads across state segments: routing-key first and second moments recover query-aligned means and variances, then separate softmaxes are mixed through a gate. This is not Mamba's state recurrence.
Example¶
Suppose a document defines x at the start and asks for it 30,000 tokens later. Full attention can address the location; a recurrent model must keep the relevant information in state. With a local window, check for global layers or extra memory rather than trusting a context-length claim alone.
Common pitfall¶
Linear complexity does not guarantee exact long-distance retrieval or real hardware speed. RoPE does not guarantee unlimited extrapolation. Check state, window, training length, and position scaling together. Local small-model results are not paper-scale evidence.
Reading order¶
Start with RoFormer's Q/K rotations, then Mamba's input-dependent state updates. Compare directly reading history with encoding history in state before moving to sparse attention, hybrids, or external memory.
- RoFormer: Enhanced Transformer with Rotary Position Embedding · Original paper
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces · Original paper
Check your understanding¶
If two models accept 32k inputs, can they recover arbitrary details from the first segment equally well?
Show suggested answer
No. Input length describes the interface or operating range. Direct addressing, window clipping, state capacity, and training distributions differ. Test retrieval with controlled positions and distractor lengths under the same budget.