Conditional computation: experts, elastic depth and width, and lookup memory¶
简体中文 · English
Foundation models / 104 questions · 26 papers
Try answering each question before opening the explanation. The bank grows as papers and questions are added.
Read this topic's comparisons, analysis, and cross-paper Q&A
104 questions · 26 papers
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
Why not give each AdaDSF layer the same token budget?
Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs
Show answer and example
Answer: Layers transform inputs by different amounts. Dense-teacher similarities allocate more to stronger transformations; routers and sparse paths then align with the teacher.
Illustrative example: A nearly identity layer may need fewer active tokens than one that reorganizes features.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Why give more compute to AdaDSF layers with lower input/output similarity?
Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs
Show answer and example
Answer: Lower cosine suggests a stronger representation transformation. Dense calibration maps layer differences to a globally corrected budget. Layer routers choose tokens and others bypass residually; allocation and alignment training serve different roles.
Illustrative example: With fixed total budget, run the more transformative layer on more tokens instead of uniformly skipping half in each.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
What controls should AdaDSF share with Uniform Mixture-of-Depths?
Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs
Show answer and example
Answer: Match global budget, teacher initialization, alignment steps, and optimizer. Vary layerwise allocation. The local negative result shows adaptation is not necessarily better.
Illustrative example: Comparing 80% and 50% retention confounds strategy with compute.
Sections in the Chinese source: 背景与主要改动, 本地复现. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Should tokens outside layer top-k be removed?
Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs
Show answer and example
Answer: No. Bypass the block with the residual and merge in original order. Correct layer ratios to the global budget. Dropping/reordering positions changes later attention and teacher alignment.
Illustrative example: Token five can skip one layer but stays at position five for the next.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
Why put patches between bytes and global attention?
Byte Latent Transformer: Patches Scale Better Than Tokens
Show answer and example
Answer: Per-byte global compute is costly. Next-byte uncertainty forms variable patches and local encoders/decoders bridge them to byte inputs and outputs.
Illustrative example: Predictable bytes can group more densely than difficult spans.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Why doesn't BLT pay equal global compute for every byte?
Byte Latent Transformer: Patches Scale Better Than Tokens
Show answer and example
Answer: Local modules handle bytes, entropy sets variable patch boundaries, and global attention operates on patches before local byte decoding. Easy spans can use longer patches. This saves global interactions, not byte handling, and is not fixed tokenization.
Illustrative example: Repeated patterns can produce fewer global patches than equally long unpredictable bytes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
How do BLT boundaries differ from fixed tokenizer boundaries?
Byte Latent Transformer: Patches Scale Better Than Tokens
Show answer and example
Answer: A fixed tokenizer uses predefined subword rules. BLT uses prefix-dependent next-byte entropy, changing compute granularity by uncertainty.
Illustrative example: The same character sequence may get different boundaries in different contexts.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Why isn't the local surprisal boundary path full BLT?
Byte Latent Transformer: Patches Scale Better Than Tokens
Show answer and example
Answer: A 512-symbol tokenizer approximates bytes with learned boundaries and shared latents, but does not implement full local encoders/decoders or 8B training. Do not label the approximate alphabet raw bytes.
Illustrative example: Boundary-aware grouping alone does not supply the complete byte-to-byte architecture.
Sections in the Chinese source: 复现边界. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
What do affinity and token error express?
Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
Show answer and example
Answer: Affinity is original router preference, while predicted error estimates quality for this token. Correct logits before top-k and supervise the head with actual next-token CE.
Illustrative example: A strongly matched expert can still be bad at that token type.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Why predict expert error in addition to affinity?
Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
Show answer and example
Answer: Affinity need not reflect expected next-token loss. An error head attenuates high-predicted-error logits before top-k and learns from actual next-token CE. Route using predictions available before the answer, not true CE.
Illustrative example: Between similar experts, predicted error can promote the more suitable one without seeing the true next token online.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
How does error-aware routing differ from load balancing?
Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
Show answer and example
Answer: Error-aware routing seeks token quality; balancing allocates compute evenly. Neither objective automatically supplies the other.
Illustrative example: Equal token counts can still send difficult cases to unsuitable experts.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Is applying the error penalty after top-k equivalent?
Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
Show answer and example
Answer: No. Pre-softmax correction can change which experts are selected. Post-selection only reweights winners. Use true next-token CE to supervise the error head, not arbitrary heuristics.
Illustrative example: A third-ranked expert can replace a high-error first choice only before the selection is final.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
Why keep the sampled token rather than replace it with a soft token?
CoTS: Competitive Token Selection for Latent Reasoning in Large Language Models
Show answer and example
Answer: Soft embeddings express alternatives but may drift from the sampled trajectory. The discrete anchor keeps at least half the weight while top-two competition controls the remainder.
Illustrative example: Close candidates raise soft mixing without removing the sampled token.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
How does CoTS preserve the sampled discrete trajectory under soft mixing?
CoTS: Competitive Token Selection for Latent Reasoning in Large Language Models
Show answer and example
Answer: Top-two competition sets the mixing fraction, capped at half, so the sampled embedding remains at least half even outside the soft set. End-of-reasoning disables soft mixing permanently; format guards can affect one step.
Illustrative example: Probabilities 0.6 and 0.4 give base mix 0.4 and keep 0.6 discrete; a larger gap weakens mixing.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
How does CoTS differ from averaging top-k embeddings?
CoTS: Competitive Token Selection for Latent Reasoning in Large Language Models
Show answer and example
Answer: Form a probability-weighted soft vector and mix it with the discrete anchor using top-two-dependent α. An unweighted mean ignores probabilities and anchoring; phase-lock controls eligibility.
Illustrative example: 0.9/0.1 and 0.5/0.5 distributions should not produce the same uniform representation.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Can soft mixing restart after the end-of-reasoning marker?
CoTS: Competitive Token Selection for Latent Reasoning in Large Language Models
Show answer and example
Answer: No. Per-example phase-lock is irreversible starting at the step emitting </think>. One-step format protection differs. Keep the discrete anchor even when it falls outside top-k.
Illustrative example: Later close probabilities do not reopen mixing after a sample finishes reasoning.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
What is the fixed top-k exposure problem?
Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts
Show answer and example
Answer: Experts just below rank k rarely receive the actual token computation and training. Elastic budgets give near-boundary experts chances before fixed-k deployment.
Illustrative example: At k=2, a frequently third-ranked expert can be useful but rarely trained on those tokens.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
How does elastic routing preserve expected compute with a random expert count?
Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts
Show answer and example
Answer: A legal symmetric support centered at k balances more and fewer experts, keeping expectation k. Boundary experts get real sparse exposure. Deployment fixes k; training load counts actual dispatch, not fixed row length.
Illustrative example: Equal probabilities for one, two, or three experts average two, but selecting three really costs more than one.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
How does elastic budget perturbation differ from raising fixed k?
Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts
Show answer and example
Answer: Raising k always spends more. Symmetric perturbation preserves expected k while changing exposure. Match parameters, initialization, and average dispatch before comparing.
Illustrative example: Equal chances of two and four experts average three, not four every time.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Why not truncate one side of the discrete Gaussian arbitrarily?
Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts
Show answer and example
Answer: Keep support symmetric around k and legal to preserve E[r]=k. Execute only selected experts, count actual loads, and fix k at inference.
Illustrative example: Near the upper limit, deleting only the high side lowers mean budget.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
Why add lookup memory for frequent local patterns?
Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
Show answer and example
Answer: Reconstructing common patterns repeatedly consumes compute. A learned n-gram table provides gated cached representations so attention and FFNs can focus on combinations.
Illustrative example: Look up a common phrase locally and compute novel combinations in the backbone.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Why is Engram conditional memory rather than another set of MoE experts?
Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
Show answer and example
Answer: MoE conditionally executes networks; Engram hashes normalized n-grams to deterministic table lookups and gates them into state. Stored patterns complement compositional compute, but memory size, collisions, bandwidth, and fusion cost remain.
Illustrative example: A common phrase can fetch a vector, but context still determines its meaning here.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
How do Engram and MoE increase different resources?
Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
Show answer and example
Answer: MoE routes compute experts. Engram reads embeddings by n-gram address. O(1) lookup still requires storing, training, and moving table values.
Illustrative example: Neural expert execution differs from fetching a phrase vector.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Should Engram replace h_t with retrieved m_t?
Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
Show answer and example
Answer: No. h′=h+sigmoid(W[h;m])⊙m preserves the residual and gates by context. Record bucket count, n-gram orders, and added-parameter training budget; preserve small-budget negative results.
Illustrative example: A poorly trained memory should not overwrite the original hidden state unconditionally.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
What carries deep information back to the next shallow step in LIFT?
Pretraining Latent Information Feedback Transformers with Teacher Supervision
Show answer and example
Answer: A continuous state accompanies the next discrete token. Training uses teacher top-k states and predicts token and next state; inference uses self-predicted states.
Illustrative example: Pass a richer distribution alongside the generated word.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Why predict next token and next continuous state in LIFT, and where does inference state come from?
Pretraining Latent Information Feedback Transformers with Teacher Supervision
Show answer and example
Answer: Training turns the teacher's top-k next-token distribution into a continuous input and jointly supervises token and state. This relays deeper signals beyond one token. Inference feeds back the model's own state, not the teacher's future distribution.
Illustrative example: A sampled word can be accompanied by other likely alternatives in state; test accumulation of model feedback errors.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
How does LIFT state feedback differ from returning only the sampled token?
Pretraining Latent Information Feedback Transformers with Teacher Supervision
Show answer and example
Answer: A token preserves one discrete choice. State carries continuous top-k distribution information through the shared embedding space. Dedicated state supervision remains necessary; adding any extra vector is not equivalent.
Illustrative example: Two nearly tied options contain information a single sampled word loses.
Sections in the Chinese source: 背景与主要改动, 本地复现. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Can the student loss update LIFT teacher state?
Pretraining Latent Information Feedback Transformers with Teacher Supervision
Show answer and example
Answer: No. Stop teacher gradients; train with token CE and forward KL on the normalized teacher top-k distribution. Inference must use the model's own state.
Illustrative example: Student errors update the student, not teacher targets to match its mistakes.
Sections in the Chinese source: 背景与主要改动, 本地复现. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
Why turn hidden states into discrete n-gram addresses?
Lngram v2: Latent N-Gram Memory with Interpretable Discrete Representations
Show answer and example
Answer: Recent-state combinations index a memory table in expected O(1); multiple results feed back into the backbone. Explicit addressing has a trainable projection and is not lookup of literal strings only.
Illustrative example: Two binary hidden-state codes can form a 2-gram key.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Why use counterfactual neighbor addresses and a zero sink with hard Lngram v2 lookups?
Lngram v2: Latent N-Gram Memory with Interpretable Discrete Representations
Show answer and example
Answer: Binary addresses are discrete and ordinary indexing gives no smooth address gradient. Neighbor comparisons train address choice. GQA reads retrieved tokens alongside a zero sink so irrelevant memory need not be injected.
Illustrative example: A neighbor may be more useful than the addressed phrase; if neither helps, attend to zero.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
How do latent addresses differ from text n-gram hashes?
Lngram v2: Latent N-Gram Memory with Interpretable Discrete Representations
Show answer and example
Answer: Backbone hidden projections carry context, while text hashes depend directly on discrete input fragments. They do not share an address space or interpretation.
Illustrative example: The same word in different contexts can yield different latent addresses.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Why include a zero sink instead of always using lookup results?
Lngram v2: Latent N-Gram Memory with Interpretable Discrete Representations
Show answer and example
Answer: It lets GQA reject irrelevant memory. Counterfactual adjacent addresses supply training signals; the integer index is not ordinarily differentiable.
Illustrative example: A query can attend to zero to keep an unhelpful lookup out of the residual.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
How does sharing weights differ from sharing runtime state?
Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers
Show answer and example
Answer: The repeated transform is the same, but KV depends on each loop's hidden inputs. LLA shares a latent while rebuilding each loop's K/V.
Illustrative example: One machine processing different intermediate materials produces different outputs.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Why can loop KV be compressed even when looped Transformer weights are shared?
Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers
Show answer and example
Answer: Loops with shared weights still generate different KV. LLA stores shared low-rank latents and reconstructs loop-specific KV. Rank and error bound savings, and reconstruction costs compute.
Illustrative example: Four iterations can store a small latent plus four maps rather than four full KV copies, subject to reconstruction quality.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
What differs between LLA KV compression and reducing parameter count?
Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers
Show answer and example
Answer: Weight sharing reduces parameter storage. Latent compression reduces history cache and adds reconstruction. Their memory scales differ.
Illustrative example: Cache keeps growing with generated length while model weights remain fixed.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
What must stay loop-specific when reconstructing KV?
Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers
Show answer and example
Answer: A shared projection produces z, then separate loop matrices U reconstruct K/V. One identical KV for all loops removes the defining difference. The small model is not a post-hoc codec or fused server kernel.
Illustrative example: Use U¹ and U² on the same z to produce two K outputs.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
Why treat layer execution as a program?
MACRO: Markov Chain Routing of Transformer Layers
Show answer and example
Answer: Context-conditioned Markov routes support skip, repeat, and residual-add rather than one fixed order. Feedback updates routes and candidate decoding chooses programs.
Illustrative example: One input can skip a layer and repeat another.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
How does MACRO change input computation without changing weights?
MACRO: Markov Chain Routing of Transformer Layers
Show answer and example
Answer: A context-conditioned routing program can skip, repeat, or residual-add layers. Feedback updates route probabilities and search selects programs; the backbone is fixed. Use validation, not isolated test gold, for selection.
Illustrative example: Two prompts can execute different paths through identical layers.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
What does routing search change compared with fine-tuning?
MACRO: Markov Chain Routing of Transformer Layers
Show answer and example
Answer: It changes how fixed layers execute, not internal parameters. Costs and generalization can differ, and a route is not a new trained backbone.
Illustrative example: Rearranging blocks differs from reshaping them.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Can MACRO greedily concatenate the highest-probability operation at each step?
MACRO: Markov Chain Routing of Transformer Layers
Show answer and example
Answer: Not generally. Top-k Viterbi preserves Markov dependencies. Locally greedy choices can miss a better full program; the classification mini-suite is not the paper's scale.
Illustrative example: A likely repeat now can reduce the score of all later reachable paths.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
What are the roles of grafting model and recipient?
Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory
Show answer and example
Answer: The grafting model supplies offline intermediate phrase states; the recipient reads this frozen memory online without rerunning the teacher for each phrase.
Illustrative example: Encode phrases once and look up values later.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Why pre-encode frequent phrases and fall back to hashed Engram?
Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory
Show answer and example
Answer: An offline model encodes common n-grams into frozen memory; the recipient tries longest-suffix exact lookup. Trainable hash memory covers misses. Projection and contextual gating decide how to use the result, separating offline knowledge cost from online access.
Illustrative example: A common trigram hits the bank while a rare combination falls back to hashing; either can be gated away.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
How does grafted memory differ from Engram trained from scratch?
Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory
Show answer and example
Answer: A pretrained model encodes and freezes the exact bank; longest-suffix lookup precedes trainable hash fallback. Table size alone is not the difference, and local results do not beat equal-parameter Engram.
Illustrative example: An absent exact phrase uses the explicit hash fallback, not a fabricated value.
Sections in the Chinese source: 背景与主要改动, 本地复现. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Which phrase state is stored in the offline bank?
Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory
Show answer and example
Answer: Freeze the final token's hidden state at the specified intermediate grafting layer. The recipient uses separate projection, context gate, and short convolution. End-to-end bank retraining changes the method.
Illustrative example: Store token four's state for a four-word phrase rather than arbitrarily averaging all four.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
What freedom do multiple residual streams add?
mHC: Manifold-Constrained Hyper-Connections
Show answer and example
Answer: They exchange information dynamically, build attention/FFN inputs, and write back to several streams. Mixing needs constraints because it can amplify depth.
Illustrative example: Keep different histories and mix as needed rather than using one cumulative vector.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Why constrain mHC residual mixing to a doubly stochastic matrix?
mHC: Manifold-Constrained Hyper-Connections
Show answer and example
Answer: Repeated unconstrained mixing can amplify deep residual signals. Nonnegative entries and unit row/column sums constrain exchanges while retaining communication. This does not bound growth of the whole nonlinear network.
Illustrative example: Two streams can mix 0.7/0.3 under both constraints; repeated unconstrained sums may grow quickly.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
Why is mHC more than another normalization layer?
mHC: Manifold-Constrained Hyper-Connections
Show answer and example
Answer: It restricts residual maps to nonnegative doubly stochastic matrices and pre/post maps to nonnegative values. Hidden normalization does not impose those operator constraints.
Illustrative example: Normalized outputs do not imply row and column sums of one.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Are unit row sums enough to validate mHC?
mHC: Manifold-Constrained Hyper-Connections
Show answer and example
Answer: No. Check nonnegativity and unit column sums as well as separate pre/post constraints. A nonexpansive residual mixer does not prove the full nonlinear model is nonexpansive or better trained.
Illustrative example: Two rows concentrating on one column satisfy row sums but fail column sums.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
Why can imbalance hurt rare specialist experts?
Optimized Deferral for Imbalanced Settings
Show answer and example
Answer: Broad experts can dominate router learning despite specialists serving particular inputs. Expert-specific margins and full training reward structure retain more information than hard minimum-cost labels.
Illustrative example: Majority-oriented training may ignore a specialist for a rare question type.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Why vary expert margins rather than add minority class weights?
Optimized Deferral for Imbalanced Settings
Show answer and example
Answer: Margins change score separation geometry; weights scale loss contributions. mild retains full training costs or rewards instead of reducing each to a winner. Estimate margins on training and never read unknown expert costs at inference.
Illustrative example: Hard minimum-cost labels can discard near-ties involving a specialist.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
What does full expert cost/reward supervision retain over argmin labels?
Optimized Deferral for Imbalanced Settings
Show answer and example
Answer: Argmin keeps only the winner. A cost-based surrogate and expert margins preserve differences and relative score constraints.
Illustrative example: A close second and an expensive second need not give identical supervision.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
May mild route using true candidate costs at inference?
Optimized Deferral for Imbalanced Settings
Show answer and example
Answer: No. Routing uses inference features only; costs and labels are unavailable. Fit margins on train and select on validation. Use stable log-sum-exp and choose the highest expert index on ties as specified.
Illustrative example: Training knows answer costs; a new question does not reveal who answers correctly.
Sections in the Chinese source: 核心公式, 代码映射与边界. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
Why select near-boundary experts early?
Rethinking Routing Exploration in Modern Mixture-of-Experts
Show answer and example
Answer: Dominant shared experts and many fine-grained combinations can obscure alternatives. Mirror routing temporarily suppresses some usual winners to train other combinations, then anneals back.
Illustrative example: Avoid always giving the same winners every early token.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Should MirrorMoE use mixed mirror probabilities to weight outputs?
Rethinking Routing Exploration in Modern Mixture-of-Experts
Show answer and example
Answer: No. Mix forward and negated-logit distributions for exploratory selection only, annealing back to standard routing. Selected outputs retain original probabilities. Track actual dispatch separately; mixing or renormalizing output weights changes the mechanism.
Illustrative example: A low-probability expert can run for exposure without receiving a larger output coefficient.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
How does mirror selection differ from arbitrary router noise?
Rethinking Routing Exploration in Modern Mixture-of-Experts
Show answer and example
Answer: Apply softmax to negated logits and mix with original probabilities using an annealing coefficient. Arbitrary noise does not reproduce its distribution or endpoint.
Illustrative example: At ρ=0, selection returns to ordinary top-k.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Which probabilities and frequencies enter MirrorMoE's load term?
Rethinking Routing Exploration in Modern Mixture-of-Experts
Show answer and example
Answer: Use actual dispatch frequencies and original ξ, not mixed p. Aggregate outputs with original weights and Z=1 without renormalizing selected experts. Selection, load, and aggregation differ.
Illustrative example: Mirror probability can change whether an expert runs, but not its original output weight.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
Why retrieve earlier depth representations at the current layer?
Multi-Stream Attention Residuals
Show answer and example
Answer: Ordinary residuals accumulate updates in one vector. MSAR makes full-width views at several time scales and retrieves across depth to access history differently.
Illustrative example: Use raw, short-window, and smoother views to form a layer input.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
How do MSAR streams differ from channel-split heads, and where does softmax apply?
Multi-Stream Attention Residuals
Show answer and example
Answer: Each stream keeps the full hidden vector at a different sequence scale. Softmax retrieves depth sources separately per stream; sigmoid combines retrieved streams without unit-sum weights. Persistent expanded views also change cost.
Illustrative example: Raw, short, and long views each contain full vectors, then independently choose historical layers.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
How does Block MSAR's persistent summary differ from caching all expanded views?
Multi-Stream Attention Residuals
Show answer and example
Answer: Store embeddings, completed block summaries, and the current partial sum only. Expand streams at read time; permanently storing them changes capacity and lifetime.
Illustrative example: Four views can serve one read without retaining four historical copies.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Is a zero source harmless before a block has a partial sum?
Multi-Stream Attention Residuals
Show answer and example
Answer: No. Zero still enters the depth softmax denominator and dilutes other sources. Omit the absent partial, and use sigmoid rather than stream softmax in the final mixer.
Illustrative example: A zero vector added to two valid sources changes their normalized weights.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
Why use recent repetition to choose byte patches?
Patch where the text is new: a compression view of where byte-level models should spend compute
Show answer and example
Answer: Byte patches reduce global calls. Recent match length captures repeatability, and a cheap two-byte predictor estimates novelty instead of relying only on entropy.
Illustrative example: A recently seen span may be reusable, whereas a novel span may need finer boundaries.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Why use matching length offline but a two-byte lookup online?
Patch where the text is new: a compression view of where byte-level models should spend compute
Show answer and example
Answer: Longer recent matches define offline novelty labels. Deployment uses a prefix-indexed table and low-count fallback instead of reading a future span. Offline labels are not available when deciding a boundary.
Illustrative example: Train with full template matches, then estimate from the two already generated bytes online.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
How does a two-byte table differ from recomputing future match length online?
Patch where the text is new: a compression view of where byte-level models should spend compute
Show answer and example
Answer: Future match lengths are offline labels. Deployment uses already generated bytes and a frozen threshold; looking past the current position leaks future information.
Illustrative example: Do not inspect whether the next 16 unseen bytes occurred before generating them.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
How does Novelty Patching handle rare byte pairs?
Patch where the text is new: a compression view of where byte-level models should spend compute
Show answer and example
Answer: Below 20 observations, back off to one-byte means, then the global mean. Calibrate τ on training bytes and reset history at document boundaries.
Illustrative example: One observed match should not determine online boundaries for a rare pair.
Sections in the Chinese source: 核心公式, 数据协议. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
What does Penelope repeatedly refine?
Penelope: Localized Latent Recurrence for Efficient Structured Reasoning
Show answer and example
Answer: A continuous hidden state at a local decoder boundary, with shared latent blocks and a gate before higher layers. It does not emit visible text on every loop.
Illustrative example: Refine one hidden position twice instead of writing the answer twice.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Why not rerun the full decoder for Penelope latent steps?
Penelope: Localized Latent Recurrence for Efficient Structured Reasoning
Show answer and example
Answer: A shared recurrent transform and gated state refine one decoder boundary repeatedly. Only that local computation increases; task gains require measurement and loop count is not reasoning ability.
Illustrative example: Refine a structured problem three times at the boundary without rerunning completed lower layers.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
How does local recurrence differ from repeating the whole decoder?
Penelope: Localized Latent Recurrence for Efficient Structured Reasoning
Show answer and example
Answer: It only repeats the designated latent block. Full reruns repeat lower and upper layers as well. Equal step counts do not match compute or information paths.
Illustrative example: Two small block calls cost less than two complete passes.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Can K independent layers replace K Penelope updates?
Penelope: Localized Latent Recurrence for Efficient Structured Reasoning
Show answer and example
Answer: No. Recurrence reuses Fθ and GRU/time gates; independent layers add parameters and alter the constraint. The local budget omits the paper's CoT-to-latent curriculum.
Illustrative example: Reusing one block twice differs from running two separately parameterized blocks.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
What extra state does Persistent Recurrent Memory add?
Persistent Recurrent Memory Between Transformer Layers Improves Language Model Generalization
Show answer and example
Answer: Between layer groups, current representations observe and update a persistent recurrent state that then influences computation through gates. It is distinct from residual accumulation.
Illustrative example: Earlier extracted information can persist and assist the subsequent layer group.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
What path changes when persistent recurrent memory sits between decoder halves?
Persistent Recurrent Memory Between Transformer Layers Improves Language Model Generalization
Show answer and example
Answer: Observe the intermediate state, update persistent state with a GRU, then gate its influence into later layers. This is neither a static prefix nor unconditional addition. Capacity and reset boundaries remain.
Illustrative example: One layer group observes a new constraint and the next can use the updated memory if relevant.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现对应. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
How does persistent GRU memory differ from a residual connection?
Persistent Recurrent Memory Between Transformer Layers Improves Language Model Generalization
Show answer and example
Answer: A residual passes representation through the current graph. Observe–update–influence maintains extra state with a separate lifetime and update rule. It is not infinite token storage.
Illustrative example: A persistent state can survive without reconstructing prior inputs word for word.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Can observe, GRU update, and gated influence be reordered?
Persistent Recurrent Memory Between Transformer Layers Improves Language Model Generalization
Show answer and example
Answer: Not without changing the topology. Specify observation source, old state, and whether later layers read pre- or post-update state; compare a reference. Local checks do not reproduce full TinyStories curves.
Illustrative example: Reading old state before updating differs from influencing with freshly updated state.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
Why can direct activation steering change expert selection?
RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models
Show answer and example
Answer: The router reads hidden state. Editing it can move expert logits and alter the computation path, confounding behavioral effects.
Illustrative example: An intended style change may also send the token to different experts.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
How does RARE steer representations without rerouting MoE tokens?
RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models
Show answer and example
Answer: Direct edits can change router-visible coordinates. Project edits into the current router's nullspace, then remove newly visible drift at protected later layers. This separates behavior edits from route changes, not all nonlinear effects.
Illustrative example: Remove a style edit's routing component initially and correct drift after intervening transformations.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
How does RARE nullspace projection differ from reducing edit strength?
RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models
Show answer and example
Answer: A smaller edit can still shift logits. Nullspace projection gives RΔh=0 for the protected linear router. Behavioral improvement still needs measurement.
Illustrative example: A tiny nonzero score change can cross top-k; the nullspace removes that direction.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Why correct later protected layers after the first projection?
RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models
Show answer and example
Answer: Nonlinear propagation can create new router-visible components. At each protected layer, project the edited-minus-clean difference into that router's nullspace.
Illustrative example: The first router can stay unchanged while the next one needs correction.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
Does depth routing choose tokens or historical layer representations?
Role-Decoupled Attention Residuals
Show answer and example
Answer: It combines residual sources from different depths, separately for QK and V, not token-level experts.
Illustrative example: Use deeper matching features for QK and shallower content for V.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Why add a separate depth route for V but retain shared Q/K routing?
Role-Decoupled Attention Residuals
Show answer and example
Answer: Q/K determine matching while V supplies retrieved content. RD-AttnRes retains the source set and attention mechanism, but learns separate depth weights for content. Routing does not create new source information.
Illustrative example: Q/K can favor deep semantics while V preserves shallow lexical details.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
What does the Block AttnRes comparison hold fixed?
Role-Decoupled Attention Residuals
Show answer and example
Answer: Residual sources and main attention stay the same; only V gets an independent depth route. This isolates role separation rather than changing the backbone.
Illustrative example: Both arms read the same layer set, with one allowing different V weights.
Sections in the Chinese source: 背景与主要改动, 本地复现. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Should Q and K also get separate routes?
Role-Decoupled Attention Residuals
Show answer and example
Answer: No. Both use h_QK, with h_V separate. Splitting Q/K adds another variable. Check actual weight divergence without treating it as guaranteed PPL improvement.
Illustrative example: Q and K project from one mixed representation, while V projects from another.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
Why treat repetition as an independent variable?
Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
Show answer and example
Answer: Same tokens can contain different unique information. Repetition can worsen expert memorization; study its interaction with sparsity and regularizers.
Illustrative example: Ten reads of 10,000 examples differ from one read of 100,000 examples.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Why can repeated data affect MoE and dense models differently at equal token counts?
Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
Show answer and example
Answer: Token count does not equal unique information. Routed experts may memorize repeated samples faster. Study stronger masking/dropout, and use held-out data; decreasing train loss is insufficient.
Illustrative example: Ten passes over one corpus differ from ten distinct corpora despite equal token budgets.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
Is matching steps enough to compare repeated-data overfitting?
Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
Show answer and example
Answer: Match or report total tokens, unique data, repetition, and sparsity. The work studies memorization under these conditions, not universal MoE inferiority.
Illustrative example: If one arm sees more repeats, architecture alone cannot explain its generalization gap.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
How should a local repeat-aware schedule be accepted?
Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
Show answer and example
Answer: Check that repeat and sparsity actually change masking/dropout according to the definition, and record applied settings. State-transition fixtures are not large pretraining or generalization evidence.
Illustrative example: Higher repetition should alter effective regularization, not just a config name.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
Why separate activation collection from the pruning algorithm?
SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
Show answer and example
Answer: The protocol collects actual autoregressive decode activations and passes them to a backend. Substituting Wanda does not implement SparseGPT's optimization.
Illustrative example: Different pruners can consume the same decode activations.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Why calibrate SparseDecoding on actual decode activations and exclude prefill?
SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
Show answer and example
Answer: Deployment generates its own prefixes, whose states differ from teacher-forced and prefill states. Collect genuine dense autoregressive activations before pruning. Collection protocol is separate from SparseGPT or Wanda backend choices.
Illustrative example: A self-generated departure from the reference can use channels that teacher-forcing calibration never sees.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
Does local Wanda use the same optimization as SparseGPT?
SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
Show answer and example
Answer: No. Wanda ranks N:M entries using weight magnitude times activation statistics, without Hessian inversion or error compensation. Sparsity does not make them equivalent.
Illustrative example: Both can keep half the weights but retain different entries and handle error differently.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Why verify bit counts and value order in 32-column sparse packs?
SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
Show answer and example
Answer: At 50% sparsity each group must contain 16 set bits and values packed in increasing retained-column order. Mismatches read wrong weights. Dense zero-multiplication is not sparse SpMV.
Illustrative example: Packed value two maps to the second set bit, not original column two.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
How does Switch increase parameters without using all experts per token?
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Show answer and example
Answer: Replace the FFN with experts and dispatch each token to one. This grows capacity without computing all expert branches, although routing and communication remain.
Illustrative example: Four experts can be available while one runs per token.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Why does top-1 Switch still need load balancing?
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Show answer and example
Answer: Top-1 caps experts per token but may concentrate traffic, bottlenecking winners and starving other experts. The auxiliary term combines actual dispatch proportions and mean router probabilities. More parameters do not imply proportional throughput.
Illustrative example: One of eight experts receiving 90% of tokens can overload despite one FFN per token.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
How does true top-1 dispatch differ from computing all outputs then one-hot masking?
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Show answer and example
Answer: Sparse dispatch executes selected experts only. Dense masking can match outputs while paying every expert's cost. Inspect actual execution.
Illustrative example: Computing four experts and discarding three is not one-expert compute.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
What do f_i and mean p_i count in Switch's load term?
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Show answer and example
Answer: f_i is actual dispatch frequency; mean p_i is average router probability. Preserve the chosen expert's gate. Local code does not cover capacity overflow or cross-device all-to-all.
Illustrative example: High dispatch and probability may correlate but are separate statistics.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
Why contract multi-turn context before routing?
SWRouter: Similarity-Contractive Window Routing for Multi-Turn Large Language Model Conversations
Show answer and example
Answer: Adjacent semantic similarity can identify repetition or unnecessary history. Construct a budgeted window first, then route on that context.
Illustrative example: Consolidate repeated clarifications before deciding what model is needed.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Why evaluate SWRouter context compression separately from model choice?
SWRouter: Similarity-Contractive Window Routing for Multi-Turn Large Language Model Conversations
Show answer and example
Answer: It first contracts conversation windows by adjacent semantic similarity, then routes. Losing a constraint can make even a good model fail; full context also does not guarantee a good route. Separate measurements locate failures.
Illustrative example: If turn two changes the budget and compression drops it, don't blame the routed model alone.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
How does semantic contraction differ from keeping the last few turns?
SWRouter: Similarity-Contractive Window Routing for Multi-Turn Large Language Model Conversations
Show answer and example
Answer: A recency window assumes newer is better; semantic contraction uses adjacent meaning to segment. Compare context quality separately from routing accuracy.
Illustrative example: An older key turn need not be removed merely because it is old.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
Is local profile-overlap routing the paper's trained contrastive encoder?
SWRouter: Similarity-Contractive Window Routing for Multi-Turn Large Language Model Conversations
Show answer and example
Answer: No. It executes segmentation and profile-overlap routing without contrastive training. Test these as separate local components, not as paper-level learned performance.
Illustrative example: Selecting from keyword overlap shows rule execution, not learned contrastive representations.
Sections in the Chinese source: 复现边界. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
Why support multiple depths in one Telescopic checkpoint?
Show answer and example
Answer: Budgets and inputs can need different depths. Supervise prefixes and full depth so shallow execution is an explicit trained option.
Illustrative example: Choose a shorter prefix rather than train one checkpoint per depth.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
Why train multiple depth prefixes instead of truncating a finished deep model?
Show answer and example
Answer: Standard training supervises the full path, not necessarily viable intermediate exits. Telescopic trains sampled prefixes with next-token supervision plus a full-depth anchor. Evaluate shallow and deep capabilities separately.
Illustrative example: Running eight versus 24 layers is supported only if the eight-layer exit was explicitly trained.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
How do sampled prefixes differ from training all depths every step?
Show answer and example
Answer: Sampled training gives prefix supervision to selected depths plus the full anchor; all depths cost more per step. Match compute rather than only parameter count.
Illustrative example: One three-layer exit is not equivalent in cost to every exit from one to 12.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
How check that random prefixes do not replace the full-depth anchor?
Show answer and example
Answer: Verify full-depth loss and gradients on every specified step and record the sampled depth. Local diagnostics do not reproduce 20B-token pretraining.
Illustrative example: Even when sampling a shallow exit, confirm full-depth parameters still receive their signal.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Basics
Why give different tokens different model widths?
WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning
Show answer and example
Answer: Static pruning uses one subnetwork for all inputs. WIDE routes attention head groups and FFN channels separately to allocate token-specific compute within layers.
Illustrative example: Simple positions can use few groups while complex evidence gets more.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Core mechanism
How is WIDE more fine-grained than skipping layers?
WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning
Show answer and example
Answer: It chooses attention-head and FFN-channel groups per token within a layer. Real savings require mask rearrangement, block skipping, and device kernels. Dense zero-masking alone pays full compute.
Illustrative example: An easy token may use fewer groups than a hard token, but only skipped execution improves latency.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Method comparison
How does per-token width routing differ from pruning half the channels permanently?
WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning
Show answer and example
Answer: Fixed pruning removes the same groups; dynamic routing selects different groups by token. Even equal average sparsity has different paths and scheduling costs.
Illustrative example: Two tokens can each use half the groups without using the same half.
Sections in the Chinese source: 背景与主要改动, 本地复现. The example illustrates the mechanism.
Foundation models / Conditional computation: experts, elastic depth and width, and lookup memory
Implementation details
How should WIDE masks interact with kernels to save work?
WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning
Show answer and example
Answer: Coordinate mask rearrangement, block skip, and on-device skipping. Multiplying a dense output by a mask does not remove matrix multiplication. Local PyTorch results are routing/sparsity/quality checks, not CUDA speed.
Illustrative example: A zero head output does not prove its QK product was skipped.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Topic guide and reading order
Start with this question¶
At a fixed parameter budget, which parameters and how many steps should each token activate?
Takeaway¶
Switch Transformer routes each token to a single expert and uses a balancing loss to avoid overloading a few experts. Engram adds learned n-gram hash lookup memory so computation need not reconstruct common patterns repeatedly. Dynamic depth, width, and recurrent steps further control compute by input. All separate total capacity from per-input activation cost.
Example¶
One expert per token out of eight increases parameter count without eight FFNs per token, but popular experts can still queue. Hash lookup for common phrases trades compute for indexing and bandwidth. Neither form of sparsity removes communication, addressing, or parameter residency costs.
Common pitfall¶
More experts do not imply proportionally more throughput. Imbalance, capacity overflow, and all-to-all communication can dominate. Lookups trade off collisions and capacity; repeated layers increase latency. Random routing diagnostics or registry labels are not capability or Evolve evidence.
Reading order¶
Read Switch's top-1 routing and balancing first, then Engram's deterministic lookup and gated fusion. Distinguish conditional computation from memory, then choose depth, width, recurrence, or routing papers for the bottleneck.
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity · Original paper
- Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models · Original paper
Check your understanding¶
If an 80B MoE activates 8B parameters per token, does it only need to store 8B?
Show suggested answer
No. Active parameters describe current computation. All routable experts still have to reside somewhere or be loaded. GPU memory, communication, and FLOPs per step are different budgets. Report total and active parameters and measured cost.