Conditional compute: choose experts, skip work, or retrieve stored knowledge?¶
简体中文 · English
Updated 2026-10-27 · 26 topic entries / 6 representative comparisons
Parameter capacity, active computation per token, and serving latency are distinct. Switch selects experts; Engram and Memory Grafting retrieve local patterns. AdaDSF, Telescopic LM, and WIDE allocate work by token and layer, full depth, and group width respectively. Identify the routing unit before discussing capacity or savings.
Mechanisms · Trade-offs · Cross-paper questions · Updates
How to read this card
The table compares representative methods. It is not a shared-benchmark leaderboard. Teaching examples are not measured results.
Mechanism comparison¶
Axes: Experts, layers, head/channel groups, or discrete memory entries · Capacity in trainable networks or frozen lookup values · Budgets per token, layer, request, or deployment depth · Whether sparse branches actually skip kernels
| Paper and source | Mechanism | Benefits and trade-offs |
|---|---|---|
| Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Activate one FFN expert per token to increase total capacity with sparse compute; an auxiliary objective balances load. | Storage, expert capacity, and cross-device communication remain. Local top-1 MoE omits overflow and all-to-all. |
| Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Hash normalized n-grams into trainable memory and context-gate the retrieved vectors into early layers. | Local patterns do not replace compositional reasoning; hash collisions and table size matter. The local trainable lookup runs, but short-budget results are not paper gains. |
| Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Encode frequent phrases offline with a pretrained model and freeze them. At serving, match the longest exact suffix or fall back to hashed Engram. | Offline bank construction amortizes knowledge acquisition, with new coverage and interface requirements. The local bank does not test scaling gains over Engram. |
| Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Allocate layer budgets from changes between dense teacher layers; each router computes only for Top-K tokens, bypassing others through residuals. | Calibrate global active ratio and align routing to the teacher. Active-token counts are not measured GPU FLOPs or latency. |
| Telescopic Language Models · Original source Sections in the Chinese source: 背景与主要改动 |
Train random depth prefixes while keeping a full-depth anchor so one checkpoint runs at several complete depths. | Deployment depth is not fine-grained per-token routing. Local checks cover random depth and anchor gradients only. |
| WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Select attention head groups and FFN channel groups separately for each token, at finer granularity than a layer. | Mask reordering and device-level block skipping are needed for speed. Dense-kernel local sparsity is not acceleration. |
Trade-off analysis¶
Expert computation and conditional memory are different capacities¶
Switch executes a nonlinear expert on the hidden input. Engram retrieves discrete local-pattern entries and gates them by context. Memory Grafting gets some entries from frozen offline pretrained representations. Sparse capacity therefore means different things for expert weights, learned tables, and frozen banks. Expected constant-time addressing does not eliminate lookup transfer and fusion costs.
Sources: Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity · Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models · Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory
State the unit of a conditional budget¶
AdaDSF sets how many tokens enter each layer; Telescopic LM runs a complete depth prefix; WIDE selects each token's attention and FFN groups. Similar active ratios conceal different dependencies, workloads, and kernels. Dense computation followed by zero masking saves no latency. Consider training compatibility and actual deployment skip granularity.
Sources: Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs · Telescopic Language Models · WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning
Cross-paper questions¶
Try answering before expanding the explanation. Each question relates at least two papers.
What do a Switch expert and an Engram or Memory Grafting lookup add for a repeated phrase?¶
Show answer and example
Answer: Switch adds a routed function executed on the current hidden input. Engram learns discrete n-gram values; Memory Grafting reuses frozen external phrase representations with fallback. Conditional computation and local lookup both help capacity but do not guarantee compositional answers.
Illustrative example: A repeated product name can retrieve a stable hint; comparing two products' uses still needs context and computation.
Sources: Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity · Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models · Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory
Why do AdaDSF, Telescopic LM, and WIDE produce different execution graphs even at half the compute?¶
Show answer and example
Answer: AdaDSF can route tokens differently across layers; Telescopic LM runs one complete prefix; WIDE routes groups within a layer per token. Equal averages do not fix slowest paths, group shapes, or memory traffic. Log token-layer assignment, actual depth, and group activation separately.
Illustrative example: Half the tokens at full depth, all tokens at half depth, and each token at half width are different forward passes.
Sources: Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs · Telescopic Language Models · WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning
Why can't Switch and WIDE speedups be inferred from mask counts?¶
Show answer and example
Answer: Experts can require dispatch and imbalance handling; grouped masks need hardware skipping with different matrix and memory patterns. Dense computation then masking only changes values. Even genuine skips require counting routing, reordering, and communication. Measure the end-to-end workload.
Illustrative example: Half the FFN channels can be slower if extra rearrangement and small matrix calls replace one efficient dense matrix.
Sources: Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity · WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning
Update log¶
Updates record new approaches, changed trade-offs, and follow-up evidence.
2026-10-27 · Initial version: compared expert execution, discrete lookup, frozen phrase grafting, and depth/width routing, separating capacity, budget units, and hardware requirements.
Covers 26 topic entries; see the full list below.
All 26 topic entries
The full catalog follows. The main comparison covers the representative papers listed above.
- Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs
- Byte Latent Transformer: Patches Scale Better Than Tokens
- Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
- CoTS: Competitive Token Selection for Latent Reasoning in Large Language Models
- Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts
- Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
- Pretraining Latent Information Feedback Transformers with Teacher Supervision
- Lngram v2: Latent N-Gram Memory with Interpretable Discrete Representations
- Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers
- MACRO: Markov Chain Routing of Transformer Layers
- Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory
- mHC: Manifold-Constrained Hyper-Connections
- Optimized Deferral for Imbalanced Settings
- Rethinking Routing Exploration in Modern Mixture-of-Experts
- Multi-Stream Attention Residuals
- Patch where the text is new: a compression view of where byte-level models should spend compute
- Penelope: Localized Latent Recurrence for Efficient Structured Reasoning
- Persistent Recurrent Memory Between Transformer Layers Improves Language Model Generalization
- RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models
- Role-Decoupled Attention Residuals
- Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
- SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- SWRouter: Similarity-Contractive Window Routing for Multi-Turn Large Language Model Conversations
- Telescopic Language Models
- WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning