Look less, later, or again: place the visual budget at the right stage¶

Updated 2026-10-27 · 16 topic entries / 6 representative comparisons

Visual budgets are more than token counts. StackTok selects encoded evidence, PACE allocates input resolution, ViMoD retains recoverable detail, Frames-on-Demand retrieves frames, AdaLoop reprocesses existing state, and OmniKVQuant lowers cache precision. Track whether evidence survives and which costs have already been paid.

Mechanisms · Trade-offs · Cross-paper questions · Updates

How to read this card

The table compares representative methods. It is not a shared-benchmark leaderboard. Teaching examples are not measured results.

Mechanism comparison¶

Axes: Before encoding, after encoding, during generation, or at cache storage · Discarding, aggregating, keeping external copies, or rereading · Question relevance, coverage, generated text, or stopping stability · Token count, peak storage, visual loops, or low-bit representations

Paper and source	Mechanism	Benefits and trade-offs
StackTok: Accelerating VLMs Inference with Budget-Adaptive Visual Token Selection · Original source Sections in the Chinese source: 背景与主要改动, 核心公式与实现对应, 复现边界	Optimize query relevance while adjusting coverage constraints with budget and switching selection criteria.	Relevance and coverage both matter. Local single-crop selector omits all five paper VLMs and is not a capability comparison.
PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	APC previews and adaptively resizes images before encoding; DDAE uses confidence-weighted language and vision attention to extract tokens.	Resizing and post-encoding selection save different stages. The real-checkpoint smoke chiefly covers APC; DDAE has formula/unit-test evidence separately.
Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 数据协议与结果, 代码映射与边界	DART retains coarse and original fine KV; TRACE holds or replaces the detail working set conditioned on causal text state.	Recoverability does not eliminate fine KV memory. Real checkpoints and CUDA gradient diagnostics cover only the models specified in the README.
Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understanding · Original source Sections in the Chinese source: 背景与主要改动, 复现边界	Build caption memory once and revisit relevant frames under a fixed budget when text evidence is insufficient.	Captions can omit details and the gate can fail to reread. Local deterministic caption/query gates do not run a long-video VLM or learned router.
AdaLoop: Thinking with the Vision Encoder · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 结果与独立对照	Modulate a shared frozen vision block by the question and loop over visual state, spending extra compute on vision.	Loops add processing but no new raw evidence. Full multi-arm training was too costly and was not reproduced; random-token gradients are diagnostic.
OmniKVQuant: KV Cache Quantization for Omni-LLMs · Original source Sections in the Chinese source: 背景与主要改动, 复现边界	Set key quantization ranges per short time window and rotate values independently by modality to lower KV precision.	Lower bits are not fewer entries. Local two-bit NumPy reference gives no GPU kernel speed evidence.

Trade-off analysis¶

When token count falls, ask where original details went¶

PACE can remove details by resizing before full encoding. StackTok selects already encoded evidence using question relevance and coverage. ViMoD retains fine KV and can restore a working set mid-generation. Frames-on-Demand keeps revisitable video and a caption index. Recoverability depends on retained evidence and an executable read path.

Sources: PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference · StackTok: Accelerating VLMs Inference with Budget-Adaptive Visual Token Selection · Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning · Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understanding

Account separately for active state, peak memory, and extra computation¶

ViMoD's small active set does not erase resident fine KV. OmniKVQuant lowers bytes per entry at some error. AdaLoop adds vision iterations for a more question-specific representation. Token reduction alone omits storage, reread, and iteration overhead.

Sources: Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning · OmniKVQuant: KV Cache Quantization for Omni-LLMs · AdaLoop: Thinking with the Vision Encoder

Cross-paper questions¶

Try answering before expanding the explanation. Each question relates at least two papers.

Does a small active ViMoD fine-KV set necessarily use less peak memory than quantized KV?¶

Show answer and example

Answer: No. Original fine KV remain for recovery, so active state differs from resident state. Quantization lowers precision but incurs rotation, quantization, and unpacking work. Compare storage tiers, peak memory, and end-to-end latency on matched input; do not multiply working-set fraction and bit width into a claimed result.

Illustrative example: One book on the desk does not mean the warehouse is empty; compressing each book saves a different resource and still costs retrieval.

Sources: Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning · OmniKVQuant: KV Cache Quantization for Omni-LLMs

Do PACE resizing, StackTok selection, and ViMoD rereads lose information at the same point?¶

Show answer and example

Answer: APC changes resolution before full encoding; StackTok selects encoded tokens; ViMoD preserves original fine KV. Information lost in low-resolution inputs cannot be recovered merely by later selection. The intervention stage determines both evidence loss and encoding already paid.

Illustrative example: Small print destroyed during resizing differs from high-resolution detail retained in an inactive cache.

Sources: PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference · StackTok: Accelerating VLMs Inference with Budget-Adaptive Visual Token Selection · Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning

Is one extra AdaLoop iteration equivalent to reading another Frames-on-Demand frame?¶

Show answer and example

Answer: No. AdaLoop transforms existing state using a question-conditioned shared block. Frame retrieval adds observations. Extra compute need not add evidence, and evidence need not be used correctly. Separate iterations, frames, and stability; gold labels cannot determine serving stops.

Illustrative example: Reanalyzing a blurry screenshot is not obtaining a clearer frame from the video.

Sources: AdaLoop: Thinking with the Vision Encoder · Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understanding · Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning

Update log¶

Updates record new approaches, changed trade-offs, and follow-up evidence.

2026-10-27 · Initial version: compared StackTok, PACE, ViMoD, on-demand frames, visual loops, and KV quantization; screened 16 abstracts and closely read six papers.

Covers 16 topic entries; see the full list below.

All 16 topic entries

The full catalog follows. The main comparison covers the representative papers listed above.

Paper-by-paper questions · All deep dives