Visual evidence budgets: tokens, frames, and on-demand revisiting¶
简体中文 · English
Multimodal models / 64 questions · 16 papers
Try answering each question before opening the explanation. The bank grows as papers and questions are added.
Read this topic's comparisons, analysis, and cross-paper Q&A
64 questions · 16 papers
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Basics
Should visual AdaLoop use identical modulation for different questions about the same image?
AdaLoop: Thinking with the Vision Encoder
Show answer and example
Answer: No. The pooled question vector conditions the visual block on the evidence each question needs. Loops add processing opportunities, not a guarantee of locating details. Shared images do not imply identical visual processing.
Illustrative example: Counting cups and reading text on them can produce different modulation vectors.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Core mechanism
Why can repeated vision loops change representations when encoder weights are frozen?
AdaLoop: Thinking with the Vision Encoder
Show answer and example
Answer: Frozen parameters do not freeze hidden state. Each loop feeds updated state into a shared block modulated by the question vector. Deep supervision trains different depths, and deployment stopping must compare predictions without reading gold.
Illustrative example: A question about objects and another about occlusion can direct the same encoder toward different details.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Method comparison
How do vision loops differ from adding language-model reasoning tokens?
AdaLoop: Thinking with the Vision Encoder
Show answer and example
Answer: Extra language steps operate in the text stream. AdaLoop spends compute on question-conditioned visual encoding with shared weights and updated visual state. Compare matched training and budgets before claiming superiority.
Illustrative example: Counting occluded objects can revisit vision features instead of only extending text reasoning after a single encoding.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Implementation details
Can stable-answer stopping inspect the gold label?
AdaLoop: Thinking with the Vision Encoder
Show answer and example
Answer: No. Deployment checks whether successive predictions agree. Training supervises several depths with positive weights summing to one. The local C interface also requires externally supplied detached draft embeddings and a fixed separator; it does not generate drafts itself.
Illustrative example: Two consecutive predictions of “three” can trigger stopping even when the true answer is “four.”
Sections in the Chinese source: 核心公式, 本地复现, 代码映射与边界. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Basics
Does caption-once mean writing a new video caption for each question?
Show answer and example
Answer: No. Build one reusable text memory, then route and budget additional vision separately for each question. Count initial captioning and per-query frame access separately; memory reuse does not mean all queries avoid visual compute.
Illustrative example: Five questions can share one caption, and only the question needing additional detail may require frames.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Core mechanism
Why add a visual-need router after building caption memory?
Show answer and example
Answer: Text remembers broad events but may omit details needed later. The router decides whether text evidence suffices and spends more vision budget only when needed. This controls evidence-acquisition cost; captions are not lossless and a routing decision is not an answer.
Illustrative example: “Someone picks up a box” can answer whether an object was taken, but reading its serial number may require looking at frames.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Method comparison
What is the trade-off versus uniformly resampling the video for every question?
Show answer and example
Answer: Uniform sampling repeatedly pays vision cost. On-demand access reuses a caption and focuses on uncertain frames, but a missed detail or routing error can hide key evidence. Compare real quality on identical questions and budgets.
Illustrative example: Ten event-order queries may share memory; a brief action in the last query still needs the right frame.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Implementation details
Do three-seed local caption/query gates validate the learned visual-need router?
Show answer and example
Answer: No. Reference state transitions check gating, top-K frame selection, and budget; there is no long-video VLM or trained semantic router. This does not prove the system can tell which questions text answers or reproduce frame savings.
Illustrative example: A rule can select at most three frames without knowing whether any contains the answer.
Sections in the Chinese source: 核心公式或操作, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Basics
What are the target and drafter roles in HAWK speculative decoding?
HAWK: Rethinking Multimodal Drafting for Speculative Decoding
Show answer and example
Answer: The cheaper drafter proposes tokens; the target verifies them and preserves the output distribution. HAWK compresses multiple target layers for a shallow drafter to improve proposals, not to bypass verification.
Illustrative example: A shallow model proposes several words, then the larger model checks how many can be accepted.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Core mechanism
Why train the HAWK drafter on shifted trajectories after its own proposals?
HAWK: Rethinking Multimodal Drafting for Speculative Decoding
Show answer and example
Answer: Proposed tokens put the system on drafter-generated prefixes. Training only on teacher prefixes hides that shift. Shifted trajectories align supervision with actual proposal states, and a mix of teacher layers conveys visual information. The target must still verify acceptance.
Illustrative example: If the first draft token chooses different wording, the second should learn the target distribution under that actual prefix.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Method comparison
How does HAWK's learned layer mix differ from using only the final target layer?
HAWK: Rethinking Multimodal Drafting for Speculative Decoding
Show answer and example
Answer: Softmax-normalized trainable weights combine hidden states from multiple layers; final-only is a fixed choice of one source. Weights sum to one but need not be equal. Mixing alone proves neither acceptance-rate gains nor speed.
Illustrative example: A high layer weight increases its share in the mixed representation rather than concatenating layers as extra tokens.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Implementation details
Does local HAWK validation include a specialized decoding kernel?
HAWK: Rethinking Multimodal Drafting for Speculative Decoding
Show answer and example
Answer: No. It checks softmax-weighted target hidden states, shifted-target supervision, and target isolation. It does not run a complete speculative verification loop or specialized kernel, so cannot demonstrate end-to-end lossless speedup.
Illustrative example: Checking that three weights sum to one does not measure accepted tokens or saved generation time.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Basics
Why count weights, KV, vision, and drafter memory together for Muse Glimmer?
Muse Glimmer 30B: An On-Device Agentic Model
Show answer and example
Answer: Device memory must hold the entire inference process. Long agent context grows KV, and vision and speculative heads add memory. A quantized language model under a size cap does not alone fit a 32GB device. Use real inputs and deployment settings.
Illustrative example: Weights below 20 GB may still leave too little space for long-trajectory KV and temporary vision tensors.
Sections in the Chinese source: 背景与主要改动, 核心公式与结构. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Core mechanism
Why is three local layers followed by one global layer different from shortening every context window?
Muse Glimmer 30B: An On-Device Agentic Model
Show answer and example
Answer: Local layers limit most interactions, while global layers still exchange distant information. GQA separately reduces KV heads. These are architectural trade-offs, not reproduction of distillation, post-training, or device performance.
Illustrative example: A distant heading can affect the current paragraph in a global layer, while local layers process adjacent sentences.
Sections in the Chinese source: 背景与主要改动, 核心公式与结构. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Method comparison
How does late-stage cross-entropy differ from fitting truncated teacher logits?
Muse Glimmer 30B: An On-Device Agentic Model
Show answer and example
Answer: Distillation uses teacher predictions, but top-k truncation drops the tail. Late cross-entropy directly supervises the target token; the paper connects the switch to truncation bias. It is not correct to describe the entire recipe as teacher matching.
Illustrative example: CE still uses a target token even if the saved teacher top-k omitted it.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Implementation details
Can the 52 layers share one attention and RoPE template?
Muse Glimmer 30B: An On-Device Agentic Model
Show answer and example
Answer: No. Repeat [Local, Local, Local, Global] 13 times with 32 query heads and two KV heads. Only local layers apply RoPE. The vision branch separately compresses 1024 patches to 256 tokens via 2×2 grouping. One layer is not the 29.6B model.
Illustrative example: Applying RoPE everywhere or alternating local and global changes the architecture even with 52 layers.
Sections in the Chinese source: 核心公式与结构, 本地复现与成本判断. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Basics
Why can one MWOP attention head have different modality-path importance?
MWOP: Modality-aware Width-wise Operation Pruning for Efficient MLLMs
Show answer and example
Answer: One head can support vision→vision, text→vision, and text→text interactions with unequal contributions. MWOP assesses and prunes paths within a head rather than treating it as indivisible.
Illustrative example: Keep text reading vision in a head that contributes little to vision-only interaction.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Core mechanism
Why re-estimate FFN channel importance after pruning attention paths?
MWOP: Modality-aware Width-wise Operation Pruning for Efficient MLLMs
Show answer and example
Answer: Pruning changes FFN input distributions, making old rankings stale. MWOP treats paths such as vision→vision and text→vision separately and prunes FFN channels by modality. Measure quality and actual speed independently.
Illustrative example: Removing text-to-vision paths can change how much fusion channels contribute.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Method comparison
What changes with modality-specific FFN pruning instead of one common channel mask?
MWOP: Modality-aware Width-wise Operation Pruning for Efficient MLLMs
Show answer and example
Answer: Text and vision tokens can retain different channel sets and capacities. One mask may hide different channel contributions. Sparse masking needs execution support; it does not guarantee hardware speedup.
Illustrative example: Text-related channels and texture channels may not overlap. Separate masks preserve both kinds, but the kernel must actually skip work.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Implementation details
Why not rank MWOP pruning by |w| rather than |w·∂L/∂w|?
MWOP: Modality-aware Width-wise Operation Pruning for Efficient MLLMs
Show answer and example
Answer: The product captures magnitude and current loss sensitivity. Magnitude alone omits sensitivity; absolute value prevents signed contributions from canceling. This local estimate depends on calibration data and path, not a guarantee deletion is harmless.
Illustrative example: Two equal weights with different gradients rank differently under the product but tie by magnitude.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Basics
Does OmniKVQuant compress weights or the current context's KV cache?
OmniKVQuant: KV Cache Quantization for Omni-LLMs
Show answer and example
Answer: It targets key/value states stored during inference, not fixed network parameters. Cache grows with context and contains different modality representations; quantized weights do not remove it.
Illustrative example: The same model has identical weights for short text and long audiovisual context, but different KV storage.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Core mechanism
Why not share one global quantization range and rotation across multimodal KV?
OmniKVQuant: KV Cache Quantization for Omni-LLMs
Show answer and example
Answer: Keys may drift over time so a global range loses local precision. Value geometry also differs by modality. Short-window ranges handle temporal shifts and modality-specific rotations handle value geometry; one static rule need not control both errors.
Illustrative example: Small audio keys can lose precision when later video keys set the global range; value structure still needs separate checks.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Method comparison
How does modality-specific rotation differ from using one matrix for all values?
OmniKVQuant: KV Cache Quantization for Omni-LLMs
Show answer and example
Answer: A shared rotation assumes one coordinate transform suits all modalities. Separate rotations allow distinct coordinate systems for audio, vision, and text. Accuracy still depends on calibration and actual distributions.
Illustrative example: Image and audio values may have different high-variance directions; one rotation might suit only one.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Implementation details
Does the local 2-bit NumPy reference demonstrate a faster GPU KV kernel?
OmniKVQuant: KV Cache Quantization for Omni-LLMs
Show answer and example
Answer: No. It checks window quantization and modality-specific transforms. Actual Triton or other kernels need real A100/A30 validation. Packing, unpacking, memory traffic, and fusion affect latency beyond bit width.
Illustrative example: Theoretically two-bit values stored and decoded in floating arrays do not yet deliver real two-bit VRAM or bandwidth savings.
Sections in the Chinese source: 核心公式或操作, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Basics
Why can't PACE infer equal vision budgets from equal image sizes?
PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference
Show answer and example
Answer: Equal-sized images can differ in global semantic density and local detail. APC estimates both off-diagonal feature similarities and top-K details that differ from background. Pixel count is not a substitute, and budget signals do not guarantee correctness.
Illustrative example: A blank wall and a densely printed page at the same resolution need not preserve the same amount of detail.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Core mechanism
Why separate image scaling and question-dependent token extraction in PACE?
PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference
Show answer and example
Answer: Scaling controls input encoding cost; extraction controls what continues downstream. Late token dropping cannot reclaim prior encoding. A shallow preview estimates difficulty, and extraction mixes language and visual attention by confidence rather than using one fixed rule.
Illustrative example: Shrink a blank image more than a dense menu, then use the question to select relevant regions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Method comparison
How does DDAE differ from an equal-weight attention average?
PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference
Show answer and example
Answer: It converts the dispersions of two normalized attention maps into confidence weights using temperature softmax. Weights can favor different streams depending on evidence. Dispersion is a method signal, not a calibrated probability of correctness.
Illustrative example: Concentrated language attention and diffuse vision attention may favor language; another question can reverse the weights.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Implementation details
Does the real-checkpoint run fully validate DDAE?
PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference
Show answer and example
Answer: No. The small real-checkpoint smoke mainly exercises APC. DDAE is checked with equations and unit tests, without the original model fork or kernels. APC excludes diagonal self-similarity and picks the top-K deviations from the local mean. Correct compression direction does not establish lossless quality.
Illustrative example: Five correct out of eight before and after compression is a small-sample observation, not verification of DDAE in generation.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Basics
What do prelude, recurrent core, and coda do in RR-VLM?
Retrofitting Recurrent Vision-Language Models from Pretrained Backbones
Show answer and example
Answer: The prelude computes context h. The core combines h with the previous state and updates s repeatedly. The coda reads the final state. They are stages, not independent voting models. Do not overwrite fixed context as though it were recurrent state.
Illustrative example: Build h once, update s₁ then s₂, and let the coda read s₂.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Core mechanism
Why can a recurrent block save stored layers without reducing executed depth?
Retrofitting Recurrent Vision-Language Models from Pretrained Backbones
Show answer and example
Answer: Repetition stores one set of weights but pays for every application. Pruning can damage language and the visual interface, so restore text, align the projector, then restore instruction behavior. Stored parameters and inference steps are different axes.
Illustrative example: Five core layers repeated three times execute 15 layers' worth of work while storing five sets of weights.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Method comparison
Why align the projector separately during staged recovery after pruning?
Retrofitting Recurrent Vision-Language Models from Pretrained Backbones
Show answer and example
Answer: Surgery can disrupt both language and vision–language alignment. Recover text first, align only the projector next, then restore instructions. The projector stage excludes KD, unlike the two surrounding recovery phases.
Illustrative example: Restore text continuation before adapting image features to avoid confounding language damage and projector mismatch.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Implementation details
What is the local status of frozen teacher and inter-loop KV sharing?
Retrofitting Recurrent Vision-Language Models from Pretrained Backbones
Show answer and example
Answer: The teacher is a separate, intact frozen copy with detached outputs; the student updates only stage-allowed parameters. Local checks verify shared calls and manual unrolling, not incremental decoding with inter-loop KV reuse or throughput.
Illustrative example: Three calls to the same core confirm parameter sharing, not that all three share one KV cache correctly.
Sections in the Chinese source: 核心公式, 本地复现, 代码映射与边界. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Basics
Is the static sink mask selected online from a new prompt?
Sink-Sparse Hybrid Attention for Efficient Inference in T2I/T2V Diffusion Transformers
Show answer and example
Answer: No. Independent dense calibration aggregates exact block mass, then takes the shortest row prefix satisfying the mass threshold. New inputs reuse the mask. This saves selection cost but depends on cross-prompt and cross-seed stability.
Illustrative example: Fixed calibration may select two regions. A new prompt's high-scoring third region does not silently redefine the mask.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Core mechanism
Why does Sink-Sparse's global branch pool sink blocks as well?
Sink-Sparse Hybrid Attention for Efficient Inference in T2I/T2V Diffusion Transformers
Show answer and example
Answer: It supplies low-resolution context over every token; the sparse branch computes fine selected interactions. They combine via their normalizers, not by adding only missing blocks. Excluding sink changes the compensation distribution, and test examples cannot reselect the offline support.
Illustrative example: The sparse branch reads the center closely while the global branch still summarizes the whole image, center included.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Method comparison
Does a full-support sparse mask alone recover dense attention?
Sink-Sparse Hybrid Attention for Efficient Inference in T2I/T2V Diffusion Transformers
Show answer and example
Answer: Not necessarily. The output still mixes a pooled global compensation branch, whose distribution may differ. Full support with gap 1 can reduce both branches to the same dense computation and is a sufficient degeneration test. General fusion is approximate.
Illustrative example: If the sparse branch sees every token but the global branch averages multiple tokens per block, their mixture can differ from dense alone.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Implementation details
Can block mass be calibrated by averaging Q/K before softmax?
Sink-Sparse Hybrid Attention for Efficient Inference in T2I/T2V Diffusion Transformers
Show answer and example
Answer: No. Compute dense token probabilities first, average over query blocks, and sum key blocks to preserve unit row mass. Softmax is nonlinear. Local Python gathers establish correctness, not speed from mask sparsity.
Illustrative example: Two queries can attend opposite ends while their mean attends the center. Early averaging loses the true mass at both ends.
Sections in the Chinese source: 核心公式, 本地复现, 成本判断与边界. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Basics
Why study which diffusion forwards to skip rather than only shrink the model?
SoftCache: Adaptive Step Size Caching for Diffusion Transformers
Show answer and example
Answer: Sampling calls the same network repeatedly; nearby states may share computation, but unconditional skipping can compound error. SoftCache adapts reuse and progress using change signals while retaining the model. Cache reliability is not a quality metric.
Illustrative example: Reuse when nearby denoising states change little and recompute when they move quickly.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Core mechanism
Why is SoftCache's step-size control more than refresh versus reuse?
SoftCache: Adaptive Step Size Caching for Diffusion Transformers
Show answer and example
Answer: Standard caches choose whether to recompute on a fixed schedule. SoftCache uses remaining staleness headroom to adjust the step and rebuild the remaining grid, so skipped nodes are not revisited. Extrapolation needs fresh anchors and early or fast-changing regions restrict jumps.
Illustrative example: Skip two of ten planned denoising nodes only if reliable, then reschedule so the same nodes are not computed later.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Method comparison
How do relative probe changes and residual extrapolation serve different purposes?
SoftCache: Adaptive Step Size Caching for Diffusion Transformers
Show answer and example
Answer: Accumulated changes estimate staleness headroom and decide refresh/step size. Two fresh residuals estimate a trend for the reused value. They are scheduling versus approximation signals and not interchangeable. One anchor cannot define a slope.
Illustrative example: The probe can say change is small without giving the current residual; the recent actual residuals estimate it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Implementation details
Can two-point extrapolation run with one fresh residual anchor?
SoftCache: Adaptive Step Size Caching for Diffusion Transformers
Show answer and example
Answer: No; one anchor gives a constant, and two anchors must be at distinct times. For t<0.1 or probe change above 0.18, do not jump. Clip step size to the endpoint and use a separate cache for each sample.
Illustrative example: The first residual has no prior timestamp to form a slope; dividing by a nonexistent time difference is invalid.
Sections in the Chinese source: 核心公式, 本地复现, 代码映射与边界. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Basics
Why not keep only the most obviously relevant StackTok positions?
StackTok: Accelerating VLMs Inference with Budget-Adaptive Visual Token Selection
Show answer and example
Answer: Some questions need several objects or a relationship. A single hotspot can consume the budget and omit necessary evidence. Coverage is not uniform sampling or unconditional diversity; the selected set should serve the question.
Illustrative example: To compare two car positions, several patches of the most prominent car are insufficient without the other car.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现对应. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Core mechanism
Why can the highest-relevance tokens be a poor fixed-budget subset?
StackTok: Accelerating VLMs Inference with Budget-Adaptive Visual Token Selection
Show answer and example
Answer: They may repeat the same region and omit scene structure. StackTok prioritizes relevance but switches criteria when a budget-dependent coverage reference indicates missing support. Coverage is not a fixed decoration; it depends on question and budget.
Illustrative example: Many face patches do not supply the doorway location needed for a layout question.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现对应. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Method comparison
How does adaptive coverage differ from a fixed mixture of two rankings?
StackTok: Accelerating VLMs Inference with Budget-Adaptive Visual Token Selection
Show answer and example
Answer: Fixed mixing holds one relevance/coverage ratio. StackTok switches criteria when current support is insufficient for the budget. This is a selection-time constraint, not a static diversity score; real fixed-budget VLM evaluation remains necessary.
Illustrative example: With two slots left and missing key regions, fill coverage; once support is sufficient, favor relevance.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现对应. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Implementation details
Does passing the local single-crop selector show acceleration in five VLMs?
StackTok: Accelerating VLMs Inference with Budget-Adaptive Visual Token Selection
Show answer and example
Answer: No. It exercises reference-gated selection on one crop, with three-seed fixtures checking ordering, gates, and budgets. No paper VLM is loaded. Correct indices do not establish a shorter forward, latency, or accuracy.
Illustrative example: Ten valid distinct indices meet the selector budget, but downstream still paying for all tokens would save no compute.
Sections in the Chinese source: 核心公式与实现对应, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Basics
Why need visual compression for hybrid linear-attention VLMs?
V-CoLA: Vision Token Compression with Linear Attention
Show answer and example
Answer: Many vision tokens consume language-layer compute and state, but linear attention stores evidence differently from softmax. V-CoLA uses actual recurrence to select what to merge or skip at deeper layers. Token count alone does not establish answer quality or inference savings.
Illustrative example: Compress redundant high-resolution background patches while preserving a small sign the question needs.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现选择. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Core mechanism
Why can't visual compression for Gated DeltaNet reuse softmax attention weights directly?
V-CoLA: Vision Token Compression with Linear Attention
Show answer and example
Answer: Linear recurrence does not expose the same full attention matrix. State updates and forgetting determine what survives. V-CoLA measures final-state readback for long-term retention and short-window response change for uniqueness, then merges contiguous blocks. These are not ordinary similarity scores.
Illustrative example: An initially salient patch may be overwritten and score poorly on final readback even if local similarity was high.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现选择. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Method comparison
What different omissions do long-term retention and short-term uniqueness address?
V-CoLA: Vision Token Compression with Linear Attention
Show answer and example
Answer: Long-term readback asks whether a token's value remains in final state. Short-term response change asks whether its update adds distinct information. Combine them 0.9/0.1 before contiguous block merging rather than deleting by static similarity alone.
Illustrative example: One token may remain useful long term while another contributes a short-lived detail; neither signal alone captures both.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现选择. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Implementation details
Why document the transposed state and query probe choices?
V-CoLA: Vision Token Compression with Linear Attention
Show answer and example
Answer: The paper uses value×key while the official backend and local code use key×value, requiring transposed recurrence operations. Equation 7 uses a query probe and the appendix shifted keys; local chooses the former and gates on actual state replay with the official kernel. Equivalence is not assumed.
Illustrative example: Reversed axes can yield legal shapes but wrong reads, so numerical replay catches errors shape checks miss.
Sections in the Chinese source: 核心公式与实现选择, 本地复现, 复现边界与 Evolve. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Basics
Why are there both intra-frame and cross-frame redundancies in video?
VETO: Video Efficient Token Optimization for Vision Language Models
Show answer and example
Answer: Nearby patches can depict one background, and adjacent frames can repeat one scene. VETO removes duplication along space and time. Each compression axis risks losing different details, so total rate alone does not show required motion or local cues survived.
Illustrative example: A wall repeats within and across frames, but a fleeting hand movement needs both temporal coverage and spatial detail.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Core mechanism
Why merge within frames before removing cross-frame redundancy?
VETO: Video Efficient Token Optimization for Vision Language Models
Show answer and example
Answer: Intra-frame merging shrinks per-frame candidates, lowering temporal matching cost. Matching all raw patches first already pays that search cost and can yield different correspondences. The order is part of the mechanism.
Illustrative example: Compress 100 patches per frame to 20 before matching adjacent frames; the reverse does not save the same work.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Method comparison
At fixed token budget, what do more frames versus more positions per frame preserve?
VETO: Video Efficient Token Optimization for Vision Language Models
Show answer and example
Answer: More frames favor temporal coverage; more positions favor spatial detail. The local tensor is frames × positions × dimensions, with each keep count bounded by its source axis and output position count equal to their product. No allocation is universally proven optimal.
Illustrative example: Four frames × 16 positions and 16 frames × four both use 64 tokens; the former sees more detail, the latter may catch brief events.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Implementation details
What does the local farthest-point representative test preserve, and what does it omit?
VETO: Video Efficient Token Optimization for Vision Language Models
Show answer and example
Answer: Deterministic farthest-point representatives test the space-first, time-second sequence and budget, and execute compression on GPU. There is no paper 7B VLM or real video benchmark. The heuristic does not reproduce semantic merging, and author throughput is not a local result.
Illustrative example: Meeting the token cap confirms both stages ran, not that small-object movement remains recognizable.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Basics
Why select evidence from independently encoded frames before passing them to a language model?
Video Encoders Built on Image Representations
Show answer and example
Answer: Framewise encoding preserves localizable evidence but produces a large sequence for long videos. Selection balances question needs, intra-frame diversity, and cross-frame links under a budget. It does not create evidence; high motion alone need not help.
Illustrative example: A question about which object was picked up first needs that object and cross-frame relations, not every changing background patch.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Core mechanism
Why refine only retained anchors without adding visual tokens?
Video Encoders Built on Image Representations
Show answer and example
Answer: Selection first preserves addressable evidence under a fixed budget. The refiner reads neighboring frames to enrich those anchors without increasing output length. Appending neighbor tokens would break the budget and confound selection with context enrichment.
Illustrative example: Ten retained patches can read neighboring motion and still output ten vectors, not ten plus 20 appended positions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Method comparison
What do intra-frame DPP and adjacent-frame correspondence rewards constrain?
Video Encoders Built on Image Representations
Show answer and example
Answer: The log-determinant balances quality and diversity to avoid near-duplicate tokens within frames. Cross-frame edges reward matched endpoints that preserve traceable relations. Similarity, margin, and spatial gates apply; edges are not guaranteed true trajectories.
Illustrative example: Select a cup and hand rather than two table patches; keep the cup's reliable match in the next frame to see movement.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Implementation details
How are tied pairs and singletons resolved, and why can the selector fill an odd budget?
Video Encoders Built on Image Representations
Show answer and example
Answer: Add a pair only if its per-token gain is strictly higher, otherwise add a singleton. Respect quotas and remaining budget. Even with negative gains or no temporal edge, select exactly B distinct indices. Local external uniform quotas and scene proxies are explicit deviations; SITE is not implemented.
Illustrative example: One remaining slot permits only a singleton; an equal per-token gain also selects the singleton.
Sections in the Chinese source: 核心公式, 本地复现, 本地结果与验收边界. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Basics
Why preview video coarsely before spending high-resolution budget?
VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs
Show answer and example
Answer: High-fidelity long-video input is costly, and relevance varies across segments and regions. A Macro Proxy supplies location cues, then detail is requested when coarse evidence is insufficient. Localization and answer evidence are not equivalent.
Illustrative example: Locate a person entering a room, then inspect their hand rather than allocating HD tokens to every segment.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现对应. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Core mechanism
Why conditionally activate Micro Tokens after a Macro Proxy locates a region?
VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs
Show answer and example
Answer: Coarse evidence cheaply removes irrelevant areas but can miss tiny text or movements. Activating detail only when needed focuses compute on uncertainty. This preserves coarse-to-fine evidence access rather than enforcing one low-resolution summary.
Illustrative example: Seeing someone hold up a sign may suffice for a yes/no question, but the number on it can require high-resolution tokens.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现对应. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Method comparison
What is the trade-off between consensus-triggered Micro and equal resolution budgets?
VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs
Show answer and example
Answer: Fixed budgets may spend on simple regions; insufficient-consensus activation focuses compute but can miss details if routing fails. Compare identical videos, questions, and cost before claiming better QA.
Illustrative example: Broad scene-type questions may not need detail; distinguishing similar tools can need it even when macro evidence seems consistent.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现对应. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Implementation details
Does the macro/micro GPU receipt validate the complete video MLLM?
VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs
Show answer and example
Answer: No. A100 executed specific state transitions, not a full video MLLM. Trigger counts, selected indices, and budgets are mechanism evidence, not LongVideoBench, throughput, or verification of semantic consensus.
Illustrative example: Supplying artificial low consensus and reaching Micro verifies branch execution, not estimating consensus from actual footage.
Sections in the Chinese source: 核心公式与实现对应, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Basics
Why recover original detail after ViMoD compresses the image?
Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning
Show answer and example
Answer: A need for small text or local evidence may emerge mid-generation. ViMoD keeps a compact overview and a recoverable detail cache, and text state selects what to read. This is neither permanent lossy compression nor reading every detail on every step; retrieval has its own costs.
Illustrative example: Recognize a receipt first, then recover the tax field when needed.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Core mechanism
Why does TRACE UPDATE replace the active Fine set rather than accumulate it?
Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning
Show answer and example
Answer: Accumulation would let active context grow beyond budget. ViMoD retains Coarse while current generation history decides whether to keep or replace Fine. This changes the attention working set, not permanent storage of recoverable Fine details.
Illustrative example: Replace the left-label working set with a right gauge while keeping the overview and ability to retrieve the label later.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Method comparison
How do DART's learned capacities differ from fixed-size pooling?
Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning
Show answer and example
Answer: Fixed pooling assigns equal position counts to all regions. DART learns region sizes and memberships under a total-capacity constraint, using continuous capacities during training. It adapts granularity but remains budgeted and cannot guarantee every detail survives.
Illustrative example: Group a large blank area together and keep more groups for dense text; a fixed grid cannot adapt that way.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual evidence budgets: tokens, frames, and on-demand revisiting
Implementation details
How do gradients cross DART groups when forward assignments are hard?
Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning
Show answer and example
Answer: Constrained capacities backpropagate through an implicit Jacobian. A hard+soft−stopgrad(soft) estimator uses hard assignments forward and Sinkhorn continuous capacities backward. Stage training needs real Fine-index labels or an online teacher and must fail if missing.
Illustrative example: A patch is assigned to group A in forward, but continuous assignments can adjust capacity backward. A hardcoded answer region is not learned retrieval evidence.
Sections in the Chinese source: 核心公式, 本地复现, 代码映射与边界. The example illustrates the mechanism.
Topic guide and reading order
Start with this question¶
When visual input is too long, which details can be skipped for now, and which need a way to be recovered?
Takeaway¶
StackTok optimizes question relevance under a coverage constraint that changes with budget, switching criteria according to the current shortfall. ViMoD's DART builds persistent coarse representations and keeps mappings to original Fine tokens; TRACE uses the generated prefix to keep or replace active details. The former mainly selects a subset before decoding; the latter supports revisiting evidence as needs change during inference.
Example¶
Listing all objects requires broad image coverage. Comparing a product label afterward requires local detail. If compression permanently deletes label tokens, later reasoning cannot recover them. Recoverable designs can fetch details but must account for Fine KV storage and retrieval. Long video has a similar trade-off between key frames and temporal coverage.
Common pitfall¶
Fewer visual tokens do not imply a proportional drop in whole-system memory or latency. Account for the visual encoder, text decoding, cache, and routing. Recoverability is not free. The local StackTok is a reference selector, while ViMoD has real-checkpoint execution and gradient verification. Neither reproduces the paper's capability gains.
Reading order¶
Start with StackTok's relevance/coverage switching for one-time budget allocation. Then compare ViMoD's Coarse/Fine and HOLD/UPDATE decisions to see why multistep reasoning may need fresh evidence. Choose additional frame selection, width pruning, or visual caching papers according to the bottleneck.
- StackTok: Accelerating VLMs Inference with Budget-Adaptive Visual Token Selection · Original paper
- Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning · Original paper
Check your understanding¶
If Fine KV is recoverable, can total visual storage count only the Coarse tokens?
Show suggested answer
No. Coarse is the resident active context; Fine KV and its mapping still require storage. Dynamic recovery also consumes bandwidth and routing compute. Report active attention budget, total cache storage, and end-to-end latency separately.