跳转至

Multimodal models by topic

简体中文 · English

Groups cover cross-modal objectives, connectors, and visual-token compression, with a separate entry per paper.

Joint code, video, and action generation

  • 2026-10-21 · Agentic World Action Models (aw0): Previous fragments become learned memory. Code normally conditions control without execution; optional hybrid executes skills only after validity, precondition and progress checks, otherwise falls back to direct actions. Not reproduced due to cost (review complete); research material only, no executable entry.

Low-resource speech translation evaluation

Frozen intermediate probes and verifier bootstrapping

  • 2026-10-08 · Unsupervised Multimodal Reward Modeling (umrm): A second experiment generates the policy's own candidates, creates success/failure pairs with a verifier, and retrains the head. Deployment utility improved while reward-benchmark scores fell.

Single-pass structured reranking

  • 2026-10-20 · Multimodal Reranking Is a System One Decision (system-one-rerank): Instead of generating a ranked list or training a separate pointwise reranker, the model scores each candidate by reading the probability of its letter label at the first answer position. The readout follows setwise and FIRST-style methods; the paper tests how this approach works zero-shot on images and visual documents, and whether text-only decision tuning transfers to visual reranking. Bounded component only; full experiments are not reproduced.

Image matching and sparse correspondence

  • 2026-10-23 · FlexGlue: Sparse-to-Dense Feature Matching and Visual Disambiguation (flexglue): It reads a local 3×3 neighborhood to avoid averaging distant modes, then refines subpixel locations with expected offsets from two axis-specific bins. A per-layer sigmoid co-visibility vote supports early rejection, trained with non-overlapping and flip-derived hard negatives rather than a gold-aware filter. Not reproduced due to cost (review complete); research material only, no executable entry.

Multi-instance editing and isolated attention

  • 2026-06-30 · Editing Everything Everywhere All at Once (mice): MICE isolates text groups in joint attention and uses a smooth spatial bias from instance masks to control visual interactions. This balances local binding and blending with the surrounding region without training a new editing model. Bounded component only; full experiments are not reproduced.

Contrastive pretraining and self-distillation

Generative auxiliary supervision for better understanding

Unified action representations and conditional attention

  • 2026-10-23 · UniAction: Unified Action Conditioning for World Models (uniaction): UniAction represents actions as sparse elements with discrete semantic IDs and continuous time/space coordinates, looks up a shared codebook and delegates position encoding to the video backbone. A group-wide log bias corrects action mass without changing the relative action logits. Bounded component only; full experiments are not reproduced.

Visual tokens and cross-modal retrieval

Visual token compression and dynamic evidence recovery

Visual state-transition supervision

  • 2026-10-08 · WOVEN: Weaving Visual World Modeling into Multimodal LLMs (woven): The mechanism is training a multimodal LM with actual visual-transition supervision, not an image scorer. The corpus contains 36,076 four-option questions, 20 scene types, 5 action types, and 8 operations: 22,728 train, 2,508 validation, 6,228 ID test, 3,496 scene-OOD, and 1,116 state-perturbation items.

Visual evidence-constrained distillation

  • 2026-10-09 · CV-OPD: Contrastive Vision On-policy Distillation (cv-opd): Three constraints matter: the positive and negative teachers differ only in their crop and see the same student prefix; CDL uses a bounded-JSD margin; and DAG uses a strict greater-than-mean token threshold over the entire retained batch. A sample is retained only if at least one rollout is correct, and then all its rollouts are retained.

Unified video embedding evaluation

  • 2026-06-12 · MVEB: Massive Video Embedding Benchmark (mveb): Classification uses an 8-shot frozen logistic probe, retrieval uses cosine and nDCG@10, and VCQA uses accuracy over candidate answers. Video and audio variants are paired and only supported modalities are run. Not reproduced due to cost (review complete); research material only, no executable entry.

Video understanding and token compression

Cross-modal connectors and frozen backbones

Audio description and contrastive ranking

  • 2026-09-01 · From Visual Cues to Spoken Narration: Rethinking Audio Description (cue2narrate-drl): DRL applies negative log-sigmoid to the score difference divided by 0.5 and enters the full loss with weight 0.2. The offline negative-caption generator is frozen, but the current model's likelihood of the generated caption still backpropagates. Bounded component only; full experiments are not reproduced.

Efficient visual token compression

  • 2025-04-07 · SmolVLM: Redefining small and efficient multimodal models (smolvlm): Move neighboring spatial tokens into channels with pixel shuffle, then project into LM space with an MLP. The paper also studies long contexts, image tiling, learned position tokens and data mixtures for 256M/500M/2.2B image/video models.