MoBA: Mixture of Block Attention for Long-Context LLMs¶

Core block-sparse routing and attention, not paper-scale context throughput.

Paper information¶

Field	Details
Paper	arXiv 2502.13189
Institution	Moonshot AI.
First public release	2025-02-18 (arXiv v1)
Author code	MoonshotAI/MoBA.
Local adapter or method key	`moba`
Local reproduction code	`src/auto_research/reproductions/moba/`

Paper method and evidence¶

Split sequences into blocks. A router chooses a few relevant blocks for each query while always retaining the current causal block.

flowchart LR
  X[History] --> B[Blocks and key centroids]
  Q[Query] --> S[Select causal blocks]
  B --> S
  S --> A[Exact attention in selected blocks]

Original paper figure¶

The original figure shows query-dependent block selection. From the original paper; copyright remains with the authors.

Core equation¶

\[ A(q)=\operatorname{softmax}(qK_{\mathcal B(q)}^\top)V_{\mathcal B(q)},\quad\mathcal B(q)=\operatorname{TopK}_b r(q,b). \]

Reported results¶

The paper approaches full-attention quality with sparse long-context training and scales to one million tokens.

Local reproduction¶

Same parameter, token, optimizer and step budget: WikiText-2, 12 steps, 64d/two layers, seed 42; llama_modern → moba PPL 421.18→423.25 (+0.49%).

auto-research reproduce --paper moba --dataset-dir data --seed 42
auto-research evolve --model micro-llm --dataset wikitext-2 --direction "Combine moba with installed paper operators" --generations 2 --population 4

Metrics. The reference uses 8-token blocks, top-2 causal blocks by query/key-centroid similarity, and exact attention over selected blocks. No fused million-context kernel.