Inference compression: weights, KV cache, and serving bandwidth¶
简体中文 · English
Foundation models / 128 questions · 32 papers
Try answering each question before opening the explanation. The bank grows as papers and questions are added.
Read this topic's comparisons, analysis, and cross-paper Q&A
128 questions · 32 papers
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why use agent phase in KV eviction?
AgentKV: Phase-Aware KV Eviction for Agentic LLMs
Show answer and example
Answer: Different phases read different history. Representatives help assess future usefulness beyond current few accesses.
Illustrative example: An early plan can regain importance during final reporting.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why keep representative queries from several agent phases for KV eviction?
AgentKV: Phase-Aware KV Eviction for Agentic LLMs
Show answer and example
Answer: Planning, tools, and reporting need different evidence. Recent queries alone may underrate material needed later. Phase representatives broaden retention scoring but cannot recover deleted KV or guarantee representativeness.
Illustrative example: Tool arguments dominate recent access, but the final summary still needs the original task constraints.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
How does a phase-aware buffer differ from a recent window?
AgentKV: Phase-Aware KV Eviction for Agentic LLMs
Show answer and example
Answer: A recent window uses time. Phase-aware scoring compares history with representatives from multiple phases. It may preserve older evidence but is limited by budget and representative quality.
Illustrative example: An old plan outside the window can still be retained when a phase query matches it.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Does local phase-aware scoring modify SGLang's real KV manager?
AgentKV: Phase-Aware KV Eviction for Agentic LLMs
Show answer and example
Answer: No. It is a scoring reference with phase and cache statistics, without SGLang or CUDA kernel changes. Real eviction, indexing, and execution still need verification.
Illustrative example: Retained indices alone do not prove GPU memory was freed and read correctly.
Sections in the Chinese source: 本地复现, 复现边界. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why use activations instead of weight magnitude alone?
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
Show answer and example
Answer: Output error depends on input activation. Calibration identifies salient channels and equivalent scaling protects them before uniform low-bit weight-only quantization.
Illustrative example: The same weight error is more consequential when multiplied by large activations.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why use equivalent scaling in AWQ rather than separately storing a few higher-precision weights?
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
Show answer and example
Answer: Calibration activations identify output-sensitive input channels. Reciprocal input/weight scaling preserves the float function while making those weights easier to represent in one low-bit format. Large weights alone do not determine sensitivity, and KV growth is unrelated.
Illustrative example: Equal weight errors have different output effects on channels with small versus large activations.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
What does AWQ add over round-to-nearest W4?
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
Show answer and example
Answer: Search activation-aware equivalent channel scales before W4 rounding. RTN uses the given scale and nearest grid. Both can retain a uniform low-bit format.
Illustrative example: Rescale an important channel before rounding rather than round original weights directly.
Sections in the Chinese source: 背景与主要改动, 本地复现. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Must input be inversely scaled when weights are scaled by S?
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
Show answer and example
Answer: Yes: XW=(XS⁻¹)(SW). Without reciprocal scaling the floating function already changes. Search scales by quantized reconstruction error; local MSE does not establish TinyChat throughput.
Illustrative example: Multiply one weight channel by two and divide its input by two.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why isn't BaKron ordinary elementwise rounding?
BaKron: Efficient Quantization with Kronecker-Factored Hessians
Show answer and example
Answer: It minimizes curvature-weighted joint error, not distance to each scalar grid point. Correlated rounding errors can amplify or cancel.
Illustrative example: Two small errors aligned in one output can be worse than a larger pair that cancels.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why use bilateral Kronecker curvature instead of input activation alone?
BaKron: Efficient Quantization with Kronecker-Factored Hessians
Show answer and example
Answer: Output-side curvature also changes the impact of weight error. A Kronecker approximation includes both sides in a quadratic objective, with structured parallel/divide-and-conquer rounding instead of a full Hessian. Independent rounding is not equivalent.
Illustrative example: Equal input errors can differ in loss sensitivity along different output directions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
What does BaKron add over input-side GPTQ-style rounding?
BaKron: Efficient Quantization with Kronecker-Factored Hessians
Show answer and example
Answer: Weight vectorized error with A⊗B to include both curvatures. Optimization must match that bilateral objective; row-wise rounding is not equivalent.
Illustrative example: Lower ordinary MSE can still incur greater bilateral weighted error.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Why check vectorization order and Kronecker factors together?
BaKron: Efficient Quantization with Kronecker-Factored Hessians
Show answer and example
Answer: Flattening convention determines which side each factor acts on. Swapped axes optimize a different error. Compare an explicit vectorized reference; random small weights are not checkpoint-scale evidence.
Illustrative example: Row- and column-major flattening require consistent factor conventions.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why retain representative queries in addition to recent ones?
Show answer and example
Answer: Long reasoning may revisit an early plan. Beacons cover diverse attention directions for budgeted retention.
Illustrative example: The last few queries can miss the original instruction needed at the end.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
How does BeaconKV preserve plans that recent-only queries would evict?
Show answer and example
Answer: Maintain a small set of representative historical beacons, score KV across phases, and protect a recent window. Representatives approximate future access, not guarantee all distant evidence survives.
Illustrative example: Ten arithmetic steps may ignore an opening constraint that the final answer needs; a beacon can keep it important.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
How does beacon scoring differ from evicting by age?
Show answer and example
Answer: Age assumes newer is better. Beacon scoring uses query-space coverage and key matches, plus recent protection. It can still miss evidence and is not a future-query oracle.
Illustrative example: An old KV matching a planning query may outlast irrelevant old entries.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Why count the Python selector and backing cache in system costs?
Show answer and example
Answer: Selection and indexing may be expensive, and retaining full backing cache does not free GPU memory. Public checkpoint tests saw slowdown and quality loss; report quality, working set, and full cost separately.
Illustrative example: A 512-item attention set can still keep all KV and spend time scoring it.
Sections in the Chinese source: 本地复现, 复现边界. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why make compressed C²KV both short and reusable by concatenation?
C²KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
Show answer and example
Answer: Short single-segment cache does not guarantee combined use. Train compression tokens and structured attention for both compression and reuse.
Illustrative example: Reuse several compressed fragments on a new query rather than prefill them again.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why jointly train compression and concatenation reuse in C²KV?
C²KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
Show answer and example
Answer: Independently compressed KV can depend on original positions or context. Compression tokens and structured attention train representations that work together across segments. This still has information loss and interface constraints.
Illustrative example: Two compressed documents should jointly supply evidence, not only reconstruct individually.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
How does joint training differ from compressing each segment and arbitrarily concatenating?
C²KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
Show answer and example
Answer: Joint training accounts for position-independent composition. Independent compression can lose cross-segment evidence. Equal compression ratios do not imply equal concatenation behavior.
Illustrative example: Two fragments can answer local questions individually yet fail cross-fragment queries when combined.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
What interface checks matter between Extractor output and concatenated input?
C²KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
Show answer and example
Answer: Verify per-segment Z shape, concatenation boundaries, and structured attention visibility. Local mini-suite omits paper-scale training and kernels; concatenability alone is insufficient.
Illustrative example: Correct lengths with wrong boundaries can cause reads from the wrong segment.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why treat KV eviction as a multistep decision?
DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference
Show answer and example
Answer: A deletion changes later available history. Local salience may not predict future loss. Stepwise full-versus-evicted KL trains a policy for later prediction.
Illustrative example: A currently unused KV can become answer evidence and may not be recoverable after deletion.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why reward predictive KL rather than token deletions in DistillCache?
DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference
Show answer and example
Answer: Deletion is a resource action; retaining predictive information is the goal. Compare full and evicted distributions per step so the policy can trade behavior loss against budget. The reference is training-only, and low KL is not universally lossless.
Illustrative example: Padding and a key ID both save one KV, but only the latter may sharply change next-token probabilities.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
Would KL-guided and minimum-cache rewards prefer the same policy?
DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference
Show answer and example
Answer: Not generally. One penalizes prediction drift; the other rewards space. Compare at fixed budget or explicit quality limits.
Illustrative example: Deleting all KV minimizes cache but can strongly change predictions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
How sanity-check DistillCache's negative KL?
DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference
Show answer and example
Answer: Equal distributions yield zero. Larger KL gives a lower reward. Use −KL(p_full||p_evict), not the reverse or a positive sign. Mini-suite does not prove policy training or deployment gains.
Illustrative example: Moving evicted predictions farther from full should not raise reward.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why score removal by output perturbation?
DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction
Show answer and example
Answer: Weight alone omits value geometry. Estimate each removal's output change and aggregate across observed queries for eviction.
Illustrative example: A high-weight value close to the output may be safe to remove.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why consider value versus current output rather than only attention weight in DropKV?
DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction
Show answer and example
Answer: Removing a term removes its value and renormalizes the rest. Change depends on p/(1−p) times output–value difference. Unusual low-weight values can matter more than redundant high-weight ones; combining single scores is approximate.
Illustrative example: Two tokens both weighted 0.1 can cause very different changes if one value equals the aggregate.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
Does exact single removal guarantee optimal multiple removal?
DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction
Show answer and example
Answer: No. Joint removal renormalizes and interacts. Per-item ranking approximates the combination, and theoretical guarantees have conditions.
Illustrative example: Two compensating values can each seem unimportant yet jointly change the output.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Does an epsilon preserve the formula when p_j rounds to one?
DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction
Show answer and example
Answer: No. p_j/(1−p_j) is singular, and local code rejects it. Check infinite recent protection, MaxPool, and averaging GQA scores into physical KV heads.
Illustrative example: Epsilon makes a finite approximation, not the exact zero-denominator result.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
How can DualQuant rescale activations without calibrating on them?
DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs
Show answer and example
Answer: Fit row/block and column scales from W alone. Apply learned column scales reciprocally to X at runtime. A solver that never reads X does not mean inference never transforms X.
Illustrative example: Derive column scales offline, then rescale input channels before quantization.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why fit both row and column scales without calibration activations?
DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs
Show answer and example
Answer: Row scaling cannot resolve shared outlier columns. Fit both to static weight reconstruction and use column scales for runtime activation rescaling. With fixed codes solve scalar least squares, project to legal scales, and requantize; code changes need not be monotone.
Illustrative example: A column large across many rows needs joint treatment and compatible input scaling.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
How does alternating closed-form scaling differ from heuristic exponent search?
DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs
Show answer and example
Answer: Separate grid assignment and scales; solve scalar least squares at fixed codes, project to legal formats, and requantize. Heuristics do not implement this optimization or its conditional descent result.
Illustrative example: A fixed-code optimal scale must be reconsidered after codes change.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Can column fitting use old codes after changing row inverse scales?
DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs
Show answer and example
Answer: No. Project row inverse scales, synchronize inner copies, and requantize before solving columns, then requantize again. The supported INT4 subset is not MXFP4.
Illustrative example: Fitting columns to stale codes violates alternating optimization.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why compress both query and document to a fixed vector count?
FLINT: Fixed-budget Late Interaction
Show answer and example
Answer: Token-level multivector indexing and search grow with length. Separate learned latents produce fixed V vectors per side for explicit cost control.
Illustrative example: Long and short documents can both yield four vectors, as can queries.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why isn't FLINT score a symmetric mean similarity at fixed V?
FLINT: Fixed-budget Late Interaction
Show answer and example
Answer: Learned latents read tokens on both sides. Score each query latent by its best document match, then sum across queries. Fixed budgets do not make pooling arbitrary or scoring symmetric.
Illustrative example: Location and time query latents each find evidence; reversing which side is summed generally changes the score.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
How does latent pooling differ from truncation or averaging?
FLINT: Fixed-budget Late Interaction
Show answer and example
Answer: Cross-attention reads all valid tokens before residuals and FFN, producing task-relevant fixed-budget vectors. Truncation loses tails and means use different aggregation.
Illustrative example: A latent can attend to a fact at the end that first-four truncation never sees.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Why define FLINT padding masks and fully masked inputs explicitly?
FLINT: Fixed-budget Late Interaction
Show answer and example
Answer: Padding cannot participate in cross-attention; local True denotes valid tokens and wholly masked rows fail. Preserve pre-LN and both residual stages, not a bare attention pool.
Illustrative example: Padding half a batch must not change what latents read.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why insert half octaves in a power-of-two grid?
Fractional Power-of-Two Quantization for Efficient and Effective Multiplier-Free LLM Inference
Show answer and example
Answer: Half-exponent spacing increases four-bit resolution, while odd exponents use 1.5 for shift-add. This trades accuracy and execution rather than allowing arbitrary floating values.
Illustrative example: A level near 1.5 is finer than only one or two.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why calibrate on the deployed 1.5 approximation to √2 in FPoT?
Fractional Power-of-Two Quantization for Efficient and Effective Multiplier-Free LLM Inference
Show answer and example
Answer: Exact half-exponent calibration followed by approximate execution adds unoptimized error. Choose scale and phase for the real shift-add grid. Integer shifts round, and activation/KV formats are separate.
Illustrative example: A code calibrated at √2×scale runs at 1.5×scale; awareness lets calibration choose a better code or scale.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
Do FPoT weights, activations, and KV use one format?
Fractional Power-of-Two Quantization for Efficient and Effective Multiplier-Free LLM Inference
Show answer and example
Answer: No. Weights use a half-octave grid; activations and KV use uniform integers. Account for each format and execution cost separately.
Illustrative example: Shift-add weight codes do not remove activation scaling rules.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Why subtract one from nonzero magnitude code m?
Fractional Power-of-Two Quantization for Efficient and Effective Multiplier-Free LLM Inference
Show answer and example
Answer: Zero has a reserved code and m=e+1 otherwise. Recover e=m−1 for parity and shift. Signed right shifts can round, so do not claim exact real multiplication by 1.5.
Illustrative example: Magnitude code one is e=0, not an odd exponent.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why share fewer K/V heads among more queries?
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Show answer and example
Answer: Queries keep distinct directions while grouped history lowers cache and read bandwidth. It interpolates between per-head MHA and single-group MQA.
Illustrative example: Four query heads can share two stored KV histories instead of four.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Does GQA reduce Q heads or KV heads, and why does it save decode memory?
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Show answer and example
Answer: Many query heads share fewer K/V heads while retaining different queries. Each historical position stores and reads fewer KV groups. This reduces independence compared with MHA, not history token count.
Illustrative example: Eight queries can use two KV groups while producing eight different attention distributions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
Which head-count axis distinguishes MHA, GQA, and MQA?
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Show answer and example
Answer: MHA generally has one KV per Q, GQA shares within groups, and MQA shares globally. Query head count can remain fixed; quality and bandwidth are empirical.
Illustrative example: Keep eight Q heads with eight, two, or one KV group.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Does temporarily expanding shared KV establish cache savings?
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Show answer and example
Answer: No. Logical broadcast can aid compute, but persistent KV must store fewer groups; retaining expanded copies loses savings. Check g(h) and distinguish local from original uptraining.
Illustrative example: Four Q heads sharing two groups should store two histories.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why expose nested widths from one checkpoint?
IrekoGPT: Turning Structured Pruning into Post-Hoc Slimmable LLMs
Show answer and example
Answer: One ordered basis supports several budget levels without separately stored pruned models. Width k takes its first k directions and receives calibration.
Illustrative example: Narrow and wide paths share the same ordered basis.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why retain projection bases for one IrekoGPT checkpoint to support nested widths?
IrekoGPT: Turning Structured Pruning into Post-Hoc Slimmable LLMs
Show answer and example
Answer: Dropping directions fixes width. Ordered bases let each width take a prefix, and multi-ratio calibration and linear correction reduce error. Shared structure does not equal shared quality or measured throughput.
Illustrative example: Switch between narrow and wide subnets only if extra basis directions were not discarded.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
How do nested widths differ from single-ratio pruning in deployment?
IrekoGPT: Turning Structured Pruning into Post-Hoc Slimmable LLMs
Show answer and example
Answer: Single-ratio pruning retains one width. Nested storage and calibration allow runtime k selection, but do not guarantee every width's quality or speed.
Illustrative example: Validating k=64 does not validate k=32 automatically.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Why not fit an independent projection basis per width?
IrekoGPT: Turning Structured Pruning into Post-Hoc Slimmable LLMs
Show answer and example
Answer: W_k=P_:kᵀWP_:k requires prefixes of one basis. Independent choices break nesting. Preserve downstream linear calibration and ridge correction; local scope is subnet diagnosis.
Illustrative example: The 32-wide basis must be the first 32 directions of the 64-wide basis.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why page long agent histories as KV?
KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU
Show answer and example
Answer: History can exceed memory and context; summaries may lose details. KVMEM stores computed KV in tiers, indexes relevant blocks, and materializes a native-window view.
Illustrative example: Many pages can exist while only several are loaded per execution.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why doesn't separating KVMEM's address space from execution window mean all-history attention?
KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU
Show answer and example
Answer: Store paged KV across tiers, retrieve relevant blocks in attention space, and materialize a bounded native-window view. Missed evidence stays out. Local full backing cache and no complete async tiering mean no VRAM or speed claim.
Illustrative example: Load a few old KV pages per query, not every page at once.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
How does paged KV reuse differ from retrieving text and prefilling it?
KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU
Show answer and example
Answer: Paged reuse materializes computed K/V; text retrieval usually computes K/V again. Position, storage, and consistency contracts differ, so retrieval count alone is not cost.
Illustrative example: Fetch cached KV versus fetch tokens and calculate their KV anew.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Why not claim lower peak VRAM while retaining full backing cache?
KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU
Show answer and example
Answer: A bounded working set does not prove offload. Local Python retains backing cache and does not establish production kernels. Report working set, residency, movement, and end-to-end time separately.
Illustrative example: Reading 512 of 2048 resident slots still occupies all 2048.
Sections in the Chinese source: 本地复现, 复现边界. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why address both repeated rounding and state outliers?
LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
Show answer and example
Answer: Recurrence accumulates quantization error and outliers dominate scale. Window buffering reduces rounding frequency; high-precision compensation separates large directions before residual quantization.
Illustrative example: Do not recompress every token or let one direction consume the entire grid.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why buffer high-precision updates in a window instead of quantizing each step?
LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
Show answer and example
Answer: Repeated rounding feeds back into recurrence. Keep a low-bit base plus precise increments and requantize at boundaries; separate compensation protects outlier directions. Buffer and compensation cost memory. Local truncated SVD is not the original kernel or throughput.
Illustrative example: Accumulate tiny updates before rounding so each is not lost individually.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
What is the trade-off between window buffers and low-bit writes every step?
LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
Show answer and example
Answer: Buffers add temporary high-precision state but reduce whole-state requantization; stepwise writes round more often. Count base, updates, and compensation.
Illustrative example: Buffering eight updates before one quantization differs from eight whole-state writes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Should the low-bit part quantize S or S−C?
LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
Show answer and example
Answer: Form C from truncated SVD, quantize S−C, then add C at readout. Quantizing full S and adding C counts it twice. Merge precise window updates at the prescribed boundaries.
Illustrative example: An outlier direction stored in C need not be stored in full in the residual too.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why connect KV error to MoE routing?
MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs
Show answer and example
Answer: Eviction perturbs attention outputs seen by the router. Attention quality alone can miss route-sensitive positions, so estimate anchors from observed instruction gates and attention.
Illustrative example: A nonmaximal KV can move a token across an expert boundary.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why reserve routing anchors in a normal MoE KV compression budget?
MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs
Show answer and example
Answer: KV deletions can change attention and subsequent routing. Propagate Fisher sensitivity from observed instruction positions through attention, reserve anchors, then fill with the base compressor. Sensitivity is a proxy, not a route-flip guarantee or permission to read gold.
Illustrative example: An older low-attention token can still affect expert choice and merit retention.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
Is Fisher trace an exact top-k router flip criterion?
MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs
Show answer and example
Answer: No. It averages distribution curvature over directions and omits the router-to-hidden Jacobian. High trace does not imply a flip.
Illustrative example: Equal trace at two positions can yield different routing changes under different perturbation directions.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Can the base compressor select anchors again when filling budget?
MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs
Show answer and example
Answer: No. Exclude already selected anchors and fill with distinct positions. Gather real KV in original temporal order. Use prefill instruction only, never future gold diagnostics.
Illustrative example: With 64 slots and 16 anchors, add 48 distinct non-anchor positions.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why assign higher precision to some RoPE pairs?
Show answer and example
Answer: Rotation planes differ in QK impact. Prefill average Q/K norm products estimate importance, and each KV head fixes its high-precision set, trading cache width for attention error.
Illustrative example: Preserve both dimensions of a high-impact rotation pair together.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why score true RoPE pairs with Q/K norm products rather than K magnitude alone?
Show answer and example
Answer: Attention error depends on both query and key. Score each rotation plane jointly to preserve coupling. Choose high-precision pairs per shared KV head during prefill and reuse them without future decode data. Memory adjacency need not be a true pair.
Illustrative example: A large key direction rarely used by queries may not deserve precision; adjacent Llama channels need not rotate together.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
Are PN-Mix 1-bit correction and FP16 residual buffering the same?
Show answer and example
Answer: No. Correction shifts 1-bit reconstruction bins without extra residual codes. The buffer holds new not-yet-quantized tokens. They have distinct numerical and storage roles.
Illustrative example: Corrected quantized groups can coexist with a new full-precision token batch.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Can Llama/SmolLM2 RoPE pairs be formed from adjacent coordinates?
Show answer and example
Answer: No for split-half layout: pair corresponding coordinates across the halves, unlike interleaved layout. Pin the prefill-selected mask across pages, without decode lookahead.
Illustrative example: Pair index i in the first half with index i in the second, not necessarily i+1.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why have full, recoverable quantized, and deleted states?
QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
Show answer and example
Answer: The middle tier keeps dormant history available with less storage and supports promotion under renewed attention, unlike deletion.
Illustrative example: A previously irrelevant window can return to high-priority reading.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why add a recoverable quantized tier between full precision and deletion?
QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
Show answer and example
Answer: Attention relevance can drift. Lower storage preserves approximate content for promotion when accumulated attention rises. Permanent deletion cannot recover it; dequantization does not restore lost precision. Local state checks are not serving performance.
Illustrative example: Demote an old clue, then read it again when a later question needs it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
Is promotion from quantized cache equivalent to never quantizing it?
QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
Show answer and example
Answer: Not necessarily. Dequantization restores access, not every original bit. Measure quantization error separately from the advantage over deletion.
Illustrative example: Converting a low-bit vector to float does not restore original decimals.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
What should QEvict verify when cumulative scores update?
QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
Show answer and example
Answer: Verify decay γ, current attention increments, promotion threshold, and slot budget. Keep actual recoverable quantized content. A log state alone does not migrate data.
Illustrative example: Crossing τ should restore the stored vector, not just increment a count.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why directly reduce native attention head width?
RACE: RoPE-Aligned Attention Compression Based on Query-Key Activation Energies
Show answer and example
Answer: Low-rank storage followed by full reconstruction saves memory but adds work. RACE folds selections and V subspaces into projections so attention runs on retained dimensions.
Illustrative example: Run four-dimensional attention directly instead of rebuilding eight dimensions.
Sections in the Chinese source: 背景与主要改动, 本地复现. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why keep original RoPE frequencies and head scaling after RACE narrows attention?
RACE: RoPE-Aligned Attention Compression Based on Query-Key Activation Energies
Show answer and example
Answer: This is a subspace of an existing model. Preserve complete matched Q/K rotation pairs, original frequencies and width scaling; fold V projections into weights for native low-dimensional compute. Calibration energy optimizes a bound, not global LM loss.
Illustrative example: Keep original second and fifth frequencies rather than renumbering them first and second.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
Why compress Q/K in pairs but V with an SVD subspace?
RACE: RoPE-Aligned Attention Compression Based on Query-Key Activation Energies
Show answer and example
Answer: Q/K preserve common RoPE planes and matching geometry. Values carry aggregated content and can use a calibrated right singular space. One common pruning rule changes their roles.
Illustrative example: Keep the corresponding K pair for each Q pair; V need not use those coordinates.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Why fold W_O when compressing V?
RACE: RoPE-Aligned Attention Compression Based on Query-Key Activation Energies
Show answer and example
Answer: W_V′=W_VP_V and W_O′=P_VᵀW_O define one subspace path. Changing V alone mismatches shape or function. Detach calibration and SVD while retaining gradients through folded forward.
Illustrative example: A four-dimensional V needs a matching output map, not the original eight-dimensional interpretation.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why omit KV importance scoring in Random Attention?
Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
Show answer and example
Answer: Scoring costs compute and can impose one shared ranking. Protect the prompt and sample generated positions uniformly per head as a simpler cache trade-off.
Illustrative example: Keep part of each head's history without scanning all tokens for importance first.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why protect full prompts but sample generated history independently per head?
Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
Show answer and example
Answer: Prompt preservation keeps task conditions and initial evidence fixed. Per-head random retention avoids scoring and gives diverse coverage, but may miss important reasoning and must fit the protected prompt budget. GPU reconstruction is not full task or kernel evaluation.
Illustrative example: Different heads can see different reasoning fragments while all still read the original question.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
How does independent per-head sampling differ from a shared random set?
Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
Show answer and example
Answer: Head-specific sets vary evidence coverage; shared sampling applies one restriction everywhere. Average budgets can match while errors and coverage correlations differ.
Illustrative example: One head can keep position five and another eight.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
May the method drop prompt entries if B is smaller than prompt length?
Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
Show answer and example
Answer: Not under full prompt protection. B−|P| must be a valid generation budget. Raise the budget or report infeasibility rather than changing the method silently.
Illustrative example: A 100-entry prompt cannot fit untouched into 80 slots.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why analyze fetch versus re-prefill break-even first?
When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
Show answer and example
Answer: Remote hits pay request and transfer costs; GPU recompute can be fast. Compare both paths by length and bandwidth, not only hit rate.
Illustrative example: Fetching a short cached prefix can be slower than recreating KV.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why can a remote KV hit be slower than re-prefill?
When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
Show answer and example
Answer: Fetch has fixed request latency and volume-dependent transfer; recompute depends on prefill throughput. Break-even depends on length and bandwidth, and asynchronous shards must avoid head-of-line blocking. Hit rate is not latency. No local remote implementation or GPU results.
Illustrative example: A short hit can spend more time opening the connection than local recompute takes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
Does B>sR with fixed T0 guarantee fetch is faster at every length?
When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
Show answer and example
Answer: No. It gives asymptotic bandwidth advantage. Compare T0+sn/B against n/R for finite requests; fixed cost can dominate short ones.
Illustrative example: Fast per-token transfer does not erase first-packet latency.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Why use per-PCP-rank remote instances and async loading?
When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
Show answer and example
Answer: MLA cache is sharded by rank; independent instances and async loads reduce head-of-line blocking. Validation needs the multi-node service and workload, not a single A100/A30 fixture.
Illustrative example: A large request on one rank should not hold a small load on another.
Sections in the Chinese source: 背景与主要改动, 成本判断. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why cache small inputs instead of writing a full matrix each token?
ReplaySSM: Cache SSM Inputs, Not State
Show answer and example
Answer: Matrix-state traffic can be bandwidth-bound. Refresh periodically and record inputs for reassociated reads, especially output-only speculative checks.
Illustrative example: Store one anchor and several small inputs instead of a whole matrix per step.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why commit only accepted inputs and keep a high-precision anchor separate from low-precision reads?
ReplaySSM: Cache SSM Inputs, Not State
Show answer and example
Answer: Replay outputs from a checkpoint and short input log without full writes every token. Rejected drafts cannot enter committed state. Refresh from the high-precision anchor and raw records to avoid rounding feedback. Reassociation preserves recurrence dependencies.
Illustrative example: Accept two of four drafts and advance only two steps; do not refresh from the rejected tail or rounded copy.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
How should output-only replay match stepwise recurrence?
ReplaySSM: Cache SSM Inputs, Not State
Show answer and example
Answer: Compute equivalent readout without materializing each state. Persistent refresh uses only committed inputs. Computation and writes change, not the reference accepted sequence.
Illustrative example: Match ordinary recurrence on accepted tokens; rejected drafts affect neither committed state.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Why preserve raw values and recompute residuals when refreshing GDN?
ReplaySSM: Cache SSM Inputs, Not State
Show answer and example
Answer: Residuals depend on the precise anchor. Reusing rounded reads or stale residuals feeds error back. Refresh from committed raw records and separately produce high/low copies; zero accepted tokens leave the anchor unchanged.
Illustrative example: Rejected drafts may have computed outputs without updating committed recurrence.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why let the quantized student generate training prefixes?
Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories
Show answer and example
Answer: Quantization changes visited reasoning states. Query the frozen teacher on the student's actual states rather than fixed full-precision answers.
Illustrative example: If the student takes a wrong turn, the teacher still scores the same prefix.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why distill on the quantized student's prefixes with per-candidate KL mixing?
Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories
Show answer and example
Answer: Quantization changes visited states, so score the student's own prefixes. Teacher/student probability ratios mix forward and reverse KL per candidate to treat over- and underestimation differently. Detach mixture weights and retain a residual mass bucket.
Illustrative example: A correct number underweighted and a connective overweighted at one position need different coefficients, not ideal teacher prefixes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
How does candidate Dynamic KL differ from one fixed mixture per response?
Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories
Show answer and example
Answer: At a prefix, some candidates are underweighted and others overweighted. α_v adapts to p_v/q_v per candidate and is detached to avoid optimizing the weight itself.
Illustrative example: Increasing support for one candidate and suppressing another need not share a mixture.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Can OnQ renormalize only the union of teacher and student top-20?
Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories
Show answer and example
Answer: No. Keep a remainder bucket with student residual gradients and teacher smoothing. Truncation would erase tail differences. OPD updates latent weights and scales, with fixed zero point.
Illustrative example: If the union covers 90%, the remaining 10% stays a class.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why assign scales by role instead of one stored axis?
RoleQ: Role-Aligned Scales for Converting Language Models to Ternary
Show answer and example
Answer: Reading and writing map different functional units to rows or columns. Scale those units independently rather than combining writer contributions.
Illustrative example: MLP input outputs and MLP output inputs represent corresponding hidden channels.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why scale reader outputs but writer inputs in RoleQ?
RoleQ: Role-Aligned Scales for Converting Language Models to Ternary
Show answer and example
Answer: Reader output units extract information; writer inputs contribute into residual space. A uniform row axis would mix functional units. Exact unweighted ternary vector solutions are not global model optima; full flow includes rotation, compensation, and distillation.
Illustrative example: One output-projection column spreads one input feature across residual dimensions and needs its own functional scale.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
How does exact ternary vector search differ from a fixed threshold?
RoleQ: Role-Aligned Scales for Converting Language Models to Ternary
Show answer and example
Answer: Search every nonempty absolute-magnitude sorted prefix for optimal support and scale under the unweighted vector objective. A threshold tries one support; local exactness is not global model optimality.
Illustrative example: A 0.7×mean threshold can miss a support included in full prefix search.
Sections in the Chinese source: 核心公式, 结果与独立对照. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Why pack and unpack ternary values and count scales?
RoleQ: Role-Aligned Scales for Converting Language Models to Ternary
Show answer and example
Answer: Theoretical trit width is not storage. Pack five trits per byte and count scales and padding; all-zero vectors use zero scale/codes. Nonzero count alone is not byte evidence.
Illustrative example: Count tail padding and per-unit scales for 77 weights instead of multiplying by log₂3.
Sections in the Chinese source: 核心公式, 结果与独立对照. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
What does RoPERelay fix after selecting sparse KV?
Compression Does Not End at Selection: Post-Selection Positional Geometry for Sparse KV Caches
Show answer and example
Answer: Retained keys still carry old position phases. Sort temporally, compact coordinates, and align subsequent question and generation without reselecting contents.
Illustrative example: Retained positions 0, 100, and 500 require new coordinates and a defined question start.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why rerotate K but not V when compacting sparse KV positions?
Compression Does Not End at Selection: Post-Selection Positional Geometry for Sparse KV Caches
Show answer and example
Answer: Undo each key's old rotation then apply its compact position; values reorder without rotation. Align unequal heads to a common right anchor and shift queries/generation. This changes coordinates, not evidence selection; scaled inverse is not simply transpose.
Illustrative example: Move old positions ten and 100 to adjacent slots, rerotate K, reorder V, and start questions at the new anchor.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
How does position reconstruction differ from prefilling selected text again?
Compression Does Not End at Selection: Post-Selection Positional Geometry for Sparse KV Caches
Show answer and example
Answer: Invert and reapply key rotations, and reorder V, without recomputing backbone representations. Re-prefill recalculates KV from text and pays a different cost.
Illustrative example: Transform existing K rather than concatenating three text spans and processing from layer one.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Why use a common M rather than per-head continuation positions?
Compression Does Not End at Selection: Post-Selection Positional Geometry for Sparse KV Caches
Show answer and example
Answer: M is the maximum kept length across layers/heads. Right-align all heads, start questions at M and answer input at M+Q. Different anchors would disagree. Invert scaled RoPE correctly.
Illustrative example: Heads with eight and four entries both start questions at eight; right-align the four retained entries.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why model attention weights and KV ownership together?
SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving
Show answer and example
Answer: TP/DP/CP/DOP assign weights, requests, and caches differently as batches and sequence lengths vary. One ownership model compares memory, communication, and switching.
Illustrative example: Four devices have different per-device memory if each request's KV is owned or sharded.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why amortize SPLASH transitions over remaining generation steps?
SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving
Show answer and example
Answer: Layouts distribute attention weights and KV ownership differently, so switching incurs transfer/handoff. Remaining savings must repay that cost under memory limits. Local work is a planner, not CUDA handoff or serving evidence.
Illustrative example: Switching can be slower with a few tokens left and useful on a long request.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
How do DOP and TP differ in simplified KV memory?
SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving
Show answer and example
Answer: TP uses W_A/T+Bks; DOP uses W_A/T+Bks/T. Both shard weights but distribute KV differently. These are capacity estimates, not measured throughput.
Illustrative example: A dominant KV term makes distribution differences more visible.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
How distinguish a transition-aware plan from executing a switch?
SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving
Show answer and example
Answer: The planner models memory and switch cost. The system must move weights/KV and preserve request state. No local CUDA handoff or B200/H200 performance evidence.
Illustrative example: Printing “switch to DOP” does not move cache ownership.
Sections in the Chinese source: 本地复现, 复现边界. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why quantize fixed-size recurrent state instead of only growing KV?
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
Show answer and example
Answer: Each request holds state and repeatedly updates it; concurrency makes fixed state costly and errors can persist. Consider lifetime and spatial block influence.
Illustrative example: Small state across many requests consumes memory, and early errors can alter later outputs.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why allocate STEPQuant bits by remaining state lifetime and key-row impact?
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
Show answer and example
Answer: Recurrence propagates errors through time, and directions affect readout differently. Allocate block precision under average bits and fit row/column scales. Local budget/error diagnostics are not throughput or renamed uniform INT8.
Illustrative example: A small direction that survives many steps may deserve more precision than a large one that soon vanishes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
How does spatiotemporal allocation differ from uniform INT8?
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
Show answer and example
Answer: Allocate more bits to high-risk blocks within a fixed average and use key-row/value-column scaling. Uniform INT8 ignores lifetime and position.
Illustrative example: A long-lived, high-impact block can get more bits than a quickly forgotten one.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Why verify actual average bits instead of the fraction of high-precision blocks?
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
Show answer and example
Answer: Width and block size both determine budget. Minimize weighted error at the actual average, and keep key-row/value-column broadcasting distinct.
Illustrative example: Average 4-bit and 8-bit blocks by element counts, not block count alone.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Does query agnostic mean no task information?
TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning
Show answer and example
Answer: No. Use guide-to-context attention but not the actual query. Task conditioning differs from query-specific selection.
Illustrative example: Know the task is summarization without knowing the exact future question.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why use task-guide attention if TaskPress is query agnostic?
TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning
Show answer and example
Answer: Guide attention supplies reusable task-level preference, combined with value scale, without the actual downstream query. Discard guide KV and retain historical positions. A small kernel does not validate guide generation or multitask gains; full run is not reproduced due to cost.
Illustrative example: Summarization guidance can compress text before a future question; reading that question first changes the task.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
Why multiply key importance and value scale?
TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning
Show answer and example
Answer: Guide attention measures read frequency and value scale captures amplitude relevant to quantization. The product accounts for matching and content; either term alone changes the method.
Illustrative example: Equal key importance can produce different scores at different value magnitudes.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Can edge average pooling divide by valid neighbors?
TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning
Show answer and example
Answer: No. Use odd K, zero padding, and fixed divisor K. Keep original position IDs after gathering, discard guide KV, and note low-index tie-breaking is local.
Illustrative example: At the first token with K=3, still divide by three even if only two neighbors are valid.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why repair retention under a fixed budget?
TwinKV: A Composable Repair Pass for KV Cache Eviction via Pairwise Key Redundancy
Show answer and example
Answer: A base policy can keep redundant keys and drop unique ones. Donor–orphan swaps address gaps without adding entries.
Illustrative example: Trade one near duplicate for unique older evidence.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why swap redundant donors for orphans rather than raise TwinKV budget?
TwinKV: A Composable Repair Pass for KV Cache Eviction via Pairwise Key Redundancy
Show answer and example
Answer: The base eviction can keep duplicates and lose unique evidence. Recover dropped keys without similar retained substitutes by removing redundant retained keys with distant twins. Protect sink/recent items. Key similarity is not semantic equivalence, and reconstruction is not generation.
Illustrative example: Replace one of two repeated explanations with a unique date at the same cache size.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
How does TwinKV compose with StreamingLLM or H2O?
TwinKV: A Composable Repair Pass for KV Cache Eviction via Pairwise Key Redundancy
Show answer and example
Answer: It repairs a wrapped policy's result rather than replacing its score. Compare the same initial policy and KV budget with repair on or off.
Illustrative example: StreamingLLM and StreamingLLM+TwinKV retain equal counts, isolating swaps.
Sections in the Chinese source: 背景与主要改动, 本地复现. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Why exclude local twins and sink/recent protected entries from donors?
TwinKV: A Composable Repair Pass for KV Cache Eviction via Pairwise Key Redundancy
Show answer and example
Answer: Find similar retained twins outside the local window and never sacrifice protected entries. Swap only min(orphans, donors) and preserve |R′|=|R|.
Illustrative example: Nearby similar keys do not automatically qualify as distant redundancy.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
What is shared between key and value caches in VFold?
VFold: Symmetry-Aware Cross-Layer Value Cache Compression
Show answer and example
Answer: Keys match historical positions; values supply content. Only aligned adjacent-layer values are merged; keys and protected first/last tokens remain separate.
Illustrative example: Two layers can share processed content while retaining independent keys.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why align adjacent Value coordinates before sharing, and why use full V for the current token?
VFold: Symmetry-Aware Cross-Layer Value Cache Compression
Show answer and example
Answer: CCA and head matching align spaces and fold transforms into V/output weights. Compute full attention for the current token before cross-layer sharing, while keys and protected ends remain independent. Invertible alignment does not make subsequent averaging lossless.
Illustrative example: Direct averaging can cancel equivalent features in different coordinates; aligned averaging is still approximate.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
How would sharing keys as well as values change VFold?
VFold: Symmetry-Aware Cross-Layer Value Cache Compression
Show answer and example
Answer: It would change position matching, not just retrieved content. The documented method keeps keys separate and shares aligned values only.
Illustrative example: Layer weights over positions can differ even when some values are shared.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Why fold inverse transforms into every relevant GQA query output slice?
VFold: Symmetry-Aware Cross-Layer Value Cache Compression
Show answer and example
Answer: Several query heads share each physical KV head. Apply inverse transforms across all matching output slices. Current token uses full V first, and protected ends add overhead for short sequences.
Illustrative example: Correcting only one of two query groups leaves the other in mismatched coordinates.
Sections in the Chinese source: 核心公式, 数据协议与结果. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
How can a quantized model recover information from the current activation's rounding residual?
ResCom: Activation Residual Compensation for Large Language Model Quantization
Show answer and example
Answer: Quantization maps a range of continuous inputs to one grid value. The forward pass can access both pre-rounding and rounded activations, so their difference retains information the quantized value lost. ResCom reads that residual through a fixed low-rank branch rather than retraining the quantizer.
Illustrative example: Example: 1.01 and 1.04 both round to 1.0. Residuals of 0.01 and 0.04 let compensation differ. This is not a model accuracy result.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why does ResCom decompose whitened output cross-statistics rather than truncate the original weights?
ResCom: Activation Residual Compensation for Large Language Model Quantization
Show answer and example
Answer: Rank should go to residual directions that occur on calibration data and affect outputs. The Gram matrix measures residual scale and correlations; error–residual cross-statistics measure output impact. Cholesky whitening then truncated SVD solves a ridge-regularized output-reconstruction objective. Direct weight truncation ignores residual distribution and solves a different problem.
Illustrative example: Example: A large residual may have little output effect while a small one is amplified. Allocate rank by output error, not residual size alone.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
What changes when a low-rank branch sees rounding residuals rather than quantized input?
ResCom: Activation Residual Compensation for Large Language Model Quantization
Show answer and example
Answer: A branch reading quantized input cannot distinguish two inputs on the same grid point. A residual branch reads the discarded component. ResCom leaves the original path frozen, unlike approaches that remove high-precision weights before quantizing the remainder.
Illustrative example: Example: Removing weights before quantization changes calibration; adding UVd+b beside a frozen path does not change its quantization settings.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Why not attach each layer's compensation as soon as its statistics are collected?
ResCom: Activation Residual Compensation for Large Language Model Quantization
Show answer and example
Answer: Early compensation changes the activations at later layers, so statistics no longer describe the same frozen model. Collect every target layer first, then install branches. Streaming merges only data from the same frozen mapping. G and C are sums: averaging without adjusting ridge changes the solution.
Illustrative example: Example: Doubling samples doubles G and C, changing a fixed ridge's relative strength. Dividing them by sample count is a different objective unless ridge is adjusted.
Sections in the Chinese source: 核心公式, 本地复现, 检查与负控. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why doesn't HOPE simply remove neurons with small weights?
Show answer and example
Answer: Scaling a ReLU neuron's input and BN scale up and its output weights down can leave the function unchanged. Raw weight magnitude is therefore unreliable. HOPE compares Hilbert-operator capacity from activation functions and output vectors under a Gaussian surrogate, not proven importance on real data.
Illustrative example: Example: Multiply the hidden activation by seven and divide output weights by seven. Predictions and functional capacity stay unchanged, although weight magnitude falls.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why is an input function times an output vector a rank-one operator?
Show answer and example
Answer: A neuron returns a scalar activation, which only changes the magnitude along one output-weight vector. The input function maps into one output direction, forming a rank-one tensor. Neuron inner products multiply an activation kernel by the output-vector inner product, not input-weight cosine alone.
Illustrative example: Example: Activation 0.2→0.8 changes 0.2w→0.8w, still along w. Changing the output vector changes functional similarity.
Sections in the Chinese source: 核心公式, 原论文关键图. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
Is HOPE Hilbert compression the same as the Hope network in Sleep?
Show answer and example
Answer: No. Hilbert Operator for Progressive Encoding analyzes frozen-network functional capacity and compresses with PRUNE, MERGE and EVICT. Sleep's Hope is a multi-frequency memory network updated in learning and offline consolidation. Local coverage is the former's kernel, pruning and one-step restoration components.
Illustrative example: Example: Scoring a frozen ResNet's neurons for removal is not teaching a language model during sleep.
Sections in the Chinese source: 背景与主要改动, 保留机制与接口, 高成本完整实验与复现边界. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Why freeze initial parameter savings and adjust downstream BN mean during pruning?
Show answer and example
Answer: The pruning priority divides distortion by initial parameter savings; recomputing from shrinking neighbor shapes changes the criterion. Remove the deleted activation's mean contribution from dense consumers' BN running means to avoid an extra shift. This does not preserve every prediction, and exact undo does not prove lossless accuracy.
Illustrative example: Example: Mean activation 0.8 and output weights [1,2] imply subtracting [0.8,1.6] from the consumer's BN mean, not adding it.
Sections in the Chinese source: 核心公式, 数据协议、结果与负控, 高成本完整实验与复现边界. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Basics
Why can't a static bias folded into a key be exact for every future query?
Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels
Show answer and example
Answer: A key shift contributes a linear function of query, whereas nonzero bias is query-independent. FoldBias fits a direction u so q·u is near 1 on context queries; future drift can increase error. It is distribution-dependent, not an identity.
Illustrative example: Example: A zero query produces zero logit change from any key shift, not bias 0.8. A query satisfying q·u=1 can express that bias exactly.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Core mechanism
Why rescale a ridge direction to μᵀu=1 rather than use unconstrained least squares?
Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels
Show answer and example
Answer: Unconstrained LS trades mean offset against projection spread and may shrink all biases. The paper rescales a ridge-stabilized inverse covariance direction to force mean projection one and minimize spread under that constraint. Covariance divides by N and ridge uses its trace; replacing N with N−1 changes the objective.
Illustrative example: Example: With one-dimensional μ=2, variance 1 and no ridge, unconstrained LS gives 2/5 and mean projection 0.8. The constrained direction 1/2 gives exactly 1.
Sections in the Chinese source: 核心公式, 检查与负控. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Method comparison
Why is FoldBias more informed than deleting bias, but not better for every query?
Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels
Show answer and example
Answer: Deletion multiplies bias by zero; folding multiplies by q·u and can be closer to one if future queries resemble calibration. Drift can be worse. The bound also needs bias range and value diameter and compares unrounded keys at fixed values, not a refitted serving stack.
Illustrative example: Example: q·u=0.9 leaves 10% bias error; q·u=−2 gives coefficient 3 vs deletion's 1. This is not LLM accuracy.
Sections in the Chinese source: 背景与主要改动, 核心公式, 论文离线与线上效果. The example illustrates the mechanism.
Foundation models / Inference compression: weights, KV cache, and serving bandwidth
Implementation details
Why actually cast keys to BF16 before refitting values?
Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels
Show answer and example
Answer: Least squares must see attention from the rounded keys. Fitting with high-precision keys then rounding changes the matrix and invalidates the intended normal equations. The local path casts before building A and returns fit-precision coefficients; cache installation, value rounding and backend correctness need separate validation.
Illustrative example: Example: BF16 maps 1.003 to 1.0. Fitting attention with 1.003 compensates an attention table that deployment will not see.
Sections in the Chinese source: 核心公式, 本地复现, 检查与负控. The example illustrates the mechanism.
Topic guide and reading order
Start with this question¶
When memory or latency is constrained, do weights, historical states, or data movement dominate?
Takeaway¶
AWQ uses calibration activations to identify important channels and applies equivalent scaling to reduce low-bit weight error. GQA shares fewer KV heads across query heads, reducing state per position. KV eviction, quantization, and tiered storage change which tokens are retained, bits per state, and where states reside, respectively. FoldBias does not choose token removals. Given retained keys and fixed bias, it encodes the bias as shared-direction shifts, preserves head width, and refits values after actual cache rounding. Compatibility and quality remain separate tests.
Example¶
Moving a prompt from 1k to 32k barely changes weight size but increases KV with sequence length and concurrency. W4 may solve loading but not ever-growing caches. Fewer KV heads, state compression, and eviction can help cache cost; check quantization error or lost evidence for the latter two.
Common pitfall¶
Smaller low-bit files do not imply proportional latency reductions: unpack kernels, bandwidth, and batch size matter. Check cache recovery and reuse. Reconstruction error, element counts, and CPU operator success cannot replace GPU throughput or long-context capability measurements.
Reading order¶
Read AWQ to see what activation-aware scaling protects, then GQA to understand query-to-KV-head sharing. After separating static weights from dynamic caches, compare eviction, cache quantization, and remote storage.
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration · Original paper
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints · Original paper
Check your understanding¶
Are AWQ and KV eviction both ways to discard unimportant historical tokens?
Show suggested answer
No. AWQ changes how static weights are represented. KV eviction removes cache positions during inference. The first affects parameter storage and numeric error; the second changes accessible history. Validate quality and resource effects separately.