Find the inference bottleneck first: weights, KV content, or cache tiers¶
简体中文 · English
Updated 2026-10-30 · 32 topic entries / 9 representative comparisons
AWQ reduces weight bit width; GQA shares KV heads; DropKV and TwinKV change retained content; KVMEM manages executable history views; Remote KV Tiers decides cross-tier loading. They can combine, but one compression ratio cannot explain quality, recoverability, and end-to-end speed. FoldBias makes compressed caches compatible with standard interfaces by folding static bias into equal-width keys, while future query drift still causes error.
Mechanisms · Trade-offs · Cross-paper questions · Updates
How to read this card
The table compares representative methods. It is not a shared-benchmark leaderboard. Teaching examples are not measured results.
Mechanism comparison¶
Axes: Weights, KV heads, tokens, or paged storage · Whether information is quantized, evicted, or retrievable externally · Activation, query, substitution similarity, or bandwidth dependencies · Separate requirements for memory, bandwidth, and recomputation savings
| Paper and source | Mechanism | Benefits and trade-offs |
|---|---|---|
| AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Use calibration activations to identify salient input channels and equivalent rescaling to protect low-bit weight reconstruction. | Changes weight storage and matmul interfaces, not history length. Local W4 MSE is not TinyChat end-to-end throughput. |
| GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Share fewer KV heads across multiple query heads, lowering per-position KV storage and reads. | Structural adaptation is usually needed and token count still grows. Small training from scratch differs from checkpoint uptraining. |
| DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 代码映射与边界 |
Score single-entry removal by output perturbation using attention probability and value difference while protecting recent history. | Individual scores are not an optimal joint eviction rule. Synthetic QKV and A100 numerical checks do not establish cache-service integration. |
| TwinKV: A Composable Repair Pass for KV Cache Eviction via Pairwise Key Redundancy · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 本地复现 |
After base eviction, swap redundant donors with irreplaceable orphans at fixed KV count. | Depends on key similarity and the base policy; repairs can regress. Preserve the negative local checkpoint reconstruction result, distinct from full generation evaluation. |
| KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Keep historical KV in tiered backing storage, retrieve blocks in model attention space, and materialize an execution view within the native window. | Not all history attends at once. Local Python retains full backing cache; no asynchronous NVMe system or memory/latency claim. |
| When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Compare remote load and recomputation, then use rank sharding and asynchronous loading to address transfer and head-of-line blocking. | Hits can be slower depending on fixed overhead and bandwidth. Paper comparisons include layout changes; local distributed experiments were not implemented due to cost. |
| ResCom: Activation Residual Compensation for Large Language Model Quantization · Original source Sections in the Chinese source: 背景与主要改动, 核心公式 |
Whiten centered residual/output error statistics, use truncated SVD, and restore the intercept. | Calibration applies to a fixed host. Quantizer and native low-bit kernels are not implemented. |
| Hilbert Operator for Progressive Encoding (HOPE): A Mathematical Framework for Deconstructing Learned Representations in Deep Networks · Original source Sections in the Chinese source: 背景与主要改动, 核心公式 |
Use a biased Gaussian–ReLU kernel and Hilbert proxy to score pruning, with small dense-layer mean compensation and one-step recovery. | Assumes a Gaussian activation proxy; no MERGE/EVICT, complete HOPE, or DEFT. Different from Sleep's Hope architecture. |
| Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 高成本完整实验 |
Fit a shared direction whose mean query projection is one, fold bias into K+√dβu, then least-squares refit V using actually bf16-rounded keys. |
Does not reduce KV heads, select AM keys, or run NNLS. Not equivalent for arbitrary future queries; the same-value unrounded bound does not directly apply after rounding or replacing values. |
Trade-off analysis¶
Four kinds of capacity reductions¶
AWQ stores the same weights in fewer bytes, GQA lowers KV heads per token, DropKV removes positions, and TwinKV reallocates a fixed position budget. The first two need not reduce history length and the latter two do not compress weights. AWQ can free room without changing long-output KV growth, while eviction may not fix weight-read bottlenecks.
Sources: AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration · GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints · DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction · TwinKV: A Composable Repair Pass for KV Cache Eviction via Pairwise Key Redundancy
Deletion versus movement changes future access¶
DropKV and TwinKV constrain visible history to what is retained. KVMEM separates the large address space from the small execution view; unselected blocks can remain in backing storage. Remote KV Tiers asks whether fixed load overhead plus bytes divided by bandwidth beats recomputation. Hits, retained data, GPU memory, and speed are distinct evidence. Local KVMEM keeps full backing cache and no remote implementation was measured.
Sources: DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction · KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU · When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
October 29: new methods and scope changes¶
ResCom adds a low-rank branch to a fixed quantized host to read activation rounding residuals; calibration and intercept must match that host. HOPE uses Gaussian–ReLU/Hilbert proxies to prune structured neurons and compensate downstream means. These are not universally equivalent pruning/quantization combinations. Local exact small components do not reproduce LLM or ImageNet matrices.
Sources: AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration · ResCom: Activation Residual Compensation for Large Language Model Quantization · Hilbert Operator for Progressive Encoding (HOPE): A Mathematical Framework for Deconstructing Learned Representations in Deep Networks
Standard cache compatibility does not eliminate compression error¶
GQA changes per-token KV storage by sharing heads. FoldBias preserves head width and query paths while folding an existing compression's static logit bias into β(q·u), accurate only when query projection is near one. Refit values after key rounding to match actual kernel inputs. AM key selection, NNLS, and engine integration remain independent; small matrices do not prove throughput or full cache quality.
Sources: Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels · GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Cross-paper questions¶
Try answering before expanding the explanation. Each question relates at least two papers.
Which memory growth terms do AWQ and GQA change in long conversations, and why might eviction still be needed?¶
Show answer and example
Answer: AWQ lowers resident weight bytes. GQA lowers per-token KV bytes but history count still grows. DropKV-like selection can bound history count while risking information loss. Report the three quantities separately.
Illustrative example: Thinner shelves do not reduce the arrival of books; GQA makes books smaller and eviction decides which books stay.
Sources: AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration · GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints · DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction
How do DropKV and TwinKV differ when two important KV entries are redundant?¶
Show answer and example
Answer: DropKV estimates each entry's output impact from observed attention probabilities and values. TwinKV repairs a base set by swapping donors with substitutes for orphans without alternatives. They address individual importance and set coverage and can combine, but require quality and budget checks.
Illustrative example: Two near-duplicate plan entries can be exchanged to recover the only authorization scope if key similarity really implies substitutability.
Sources: DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction · TwinKV: A Composable Repair Pass for KV Cache Eviction via Pairwise Key Redundancy
Why might Remote KV Tiers recommend recomputation after KVMEM finds a block?¶
Show answer and example
Answer: An address makes reuse possible, not free. Low bandwidth, small blocks, or queueing can make T0 + bytes/bandwidth exceed a new prefill. Execution view limits visibility; the break-even rule decides how to retrieve. Hit rate alone cannot choose.
Illustrative example: A remote hit on a short prefix may lose to local recomputation, while a long prefix amortizes fixed overhead.
Sources: KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU · When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
Why can't ResCom correction rank be read as the number of HOPE neurons that can be pruned?¶
Show answer and example
Answer: ResCom budgets low-rank output-error compensation. HOPE costs depend on neuron Hilbert capacity and original parameter footprint. Their parameter locations, states, and calibration assumptions differ.
Illustrative example: Keeping every quantized output plus a rank-two correction is not deleting two neurons and compensating the downstream mean.
Sources: AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration · ResCom: Activation Residual Compensation for Large Language Model Quantization · Hilbert Operator for Progressive Encoding (HOPE): A Mathematical Framework for Deconstructing Learned Representations in Deep Networks
Why check future query quality when FoldBias preserves head width and works with a standard kernel?¶
Show answer and example
Answer: Folded bias is β(q·u), not always β. The mean constraint μ·u=1 only fits the context distribution; future queries can drift. GQA head sharing compresses another object. Verify compatibility, memory, and quality separately.
Illustrative example: With a future projection of 0.6, only about 60% of the original bias remains despite valid tensor shapes.
Sources: Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels · GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Update log¶
Updates record new approaches, changed trade-offs, and follow-up evidence.
2026-10-30 · Reviewed FoldBias's constrained direction, rank-one key displacement, bf16 before value refitting, and same-value bound. Distinguished interface compatibility from structural GQA compression and retained future query drift caveats.
Affected: Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels · GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
2026-10-29 · ResCom adds low-rank activation-rounding residual correction to a fixed quantized host, with host-matched calibration and intercept. HOPE assesses structured neuron pruning with Gaussian–ReLU/Hilbert proxies and compensates downstream means. Their small exact local components do not guarantee equivalent combined compression or cover LLM/ImageNet capability matrices.
Affected: ResCom: Activation Residual Compensation for Large Language Model Quantization · Hilbert Operator for Progressive Encoding (HOPE): A Mathematical Framework for Deconstructing Learned Representations in Deep Networks
2026-10-27 · Initial version: compared low-bit weights, shared KV heads, perturbation eviction, fixed-budget repair, historical execution views, and remote loading across capacity, recoverability, and latency.
Covers 29 topic entries; see the full list below.
All 32 topic entries
The full catalog follows. The main comparison covers the representative papers listed above.
- AgentKV: Phase-Aware KV Eviction for Agentic LLMs
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- BaKron: Efficient Quantization with Kronecker-Factored Hessians
- BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference
- C²KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
- DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference
- DropKV: Decoupling Residual-Output Perturbation for Near-Optimal KV-Cache Eviction
- DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs
- FLINT: Fixed-budget Late Interaction
- Fractional Power-of-Two Quantization for Efficient and Effective Multiplier-Free LLM Inference
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- IrekoGPT: Turning Structured Pruning into Post-Hoc Slimmable LLMs
- KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU
- LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
- MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs
- PN-Mix: Pair-Norm Guided Mixed-Precision Key Cache Quantization for Efficient Long-Output LLM Inference
- QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
- RACE: RoPE-Aligned Attention Compression Based on Query-Key Activation Energies
- Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
- When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
- ReplaySSM: Cache SSM Inputs, Not State
- Right On Q: Recovering Reasoning under Quantization via Self-Generated Trajectories
- RoleQ: Role-Aligned Scales for Converting Language Models to Ternary
- Compression Does Not End at Selection: Post-Selection Positional Geometry for Sparse KV Caches
- SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving
- STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
- TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning
- TwinKV: A Composable Repair Pass for KV Cache Eviction via Pairwise Key Redundancy
- VFold: Symmetry-Aware Cross-Layer Value Cache Compression
- ResCom: Activation Residual Compensation for Large Language Model Quantization
- Hilbert Operator for Progressive Encoding (HOPE): A Mathematical Framework for Deconstructing Learned Representations in Deep Networks
- Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels