跳转至

Multimodal models by year

简体中文 · English

Browse by first arXiv v1 publication year, newest first within each year.

2026

  • 2026-10 · FlexGlue: Sparse-to-Dense Feature Matching and Visual Disambiguation (flexglue): It reads a local 3×3 neighborhood to avoid averaging distant modes, then refines subpixel locations with expected offsets from two axis-specific bins. A per-layer sigmoid co-visibility vote supports early rejection, trained with non-overlapping and flip-derived hard negatives rather than a gold-aware filter. Not reproduced due to cost (review complete); research material only, no executable entry.
  • 2026-10 · UniAction: Unified Action Conditioning for World Models (uniaction): UniAction represents actions as sparse elements with discrete semantic IDs and continuous time/space coordinates, looks up a shared codebook and delegates position encoding to the video backbone. A group-wide log bias corrects action mass without changing the relative action logits. Bounded component only; full experiments are not reproduced.
  • 2026-10 · Agentic World Action Models (aw0): Previous fragments become learned memory. Code normally conditions control without execution; optional hybrid executes skills only after validity, precondition and progress checks, otherwise falls back to direct actions. Not reproduced due to cost (review complete); research material only, no executable entry.
  • 2026-10 · Multimodal Reranking Is a System One Decision (system-one-rerank): Instead of generating a ranked list or training a separate pointwise reranker, the model scores each candidate by reading the probability of its letter label at the first answer position. The readout follows setwise and FIRST-style methods; the paper tests how this approach works zero-shot on images and visual documents, and whether text-only decision tuning transfers to visual reranking. Bounded component only; full experiments are not reproduced.
  • 2026-10 · CV-OPD: Contrastive Vision On-policy Distillation (cv-opd): Three constraints matter: the positive and negative teachers differ only in their crop and see the same student prefix; CDL uses a bounded-JSD margin; and DAG uses a strict greater-than-mean token threshold over the entire retained batch. A sample is retained only if at least one rollout is correct, and then all its rollouts are retained.
  • 2026-10 · Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning (vimod): TRACE uses causal hidden states from already-generated text to hold or replace the working set. Joint-KV restores original details on demand.
  • 2026-10 · Unsupervised Multimodal Reward Modeling (umrm): A second experiment generates the policy's own candidates, creates success/failure pairs with a verifier, and retrains the head. Deployment utility improved while reward-benchmark scores fell.
  • 2026-10 · V-CoLA: Vision Token Compression with Linear Attention (vcola): Partition contiguous tokens by cumulative importance and merge them by weight; remove visual tokens at a deeper layer. See Section 3 and Figure 5. Method reference 1
  • 2026-10 · WOVEN: Weaving Visual World Modeling into Multimodal LLMs (woven): The mechanism is training a multimodal LM with actual visual-transition supervision, not an image scorer. The corpus contains 36,076 four-option questions, 20 scene types, 5 action types, and 8 operations: 22,728 train, 2,508 validation, 6,228 ID test, 3,496 scene-OOD, and 1,116 state-perturbation items.
  • 2026-10 · EmbeddingGemma 2: Unified and Efficient Multimodal Representations (embedding-gemma2): The technical paper reports 744M total parameters: 271M in the text backbone including the embedder, 167M in vision, and 305M in audio. The authors give 439M for the image–text path; these are rounded approximations. Checkpoint inference; not a reproduction of training.
  • 2026-09 · StackTok: Accelerating VLMs Inference with Budget-Adaptive Visual Token Selection (stacktok): Use query relevance as the objective and coverage as a budget-dependent support constraint, progressively switching between the two selection criteria.
  • 2026-09 · VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs (videomm): Locate relevant regions on a low-resolution Macro Proxy, then activate high-fidelity Micro Tokens only if macro-level consensus is insufficient.
  • 2026-09 · From Visual Cues to Spoken Narration: Rethinking Audio Description (cue2narrate-drl): DRL applies negative log-sigmoid to the score difference divided by 0.5 and enters the full loss with weight 0.2. The offline negative-caption generator is frozen, but the current model's likelihood of the generated caption still backpropagates. Bounded component only; full experiments are not reproduced.
  • 2026-08 · Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction (gas): Generation gradients improve shared visual features without directly changing the understanding-only upper layers. Generation can be removed entirely at deployment.
  • 2026-06 · Editing Everything Everywhere All at Once (mice): MICE isolates text groups in joint attention and uses a smooth spatial bias from instance masks to control visual interactions. This balances local binding and blending with the surrounding region without training a new editing model. Bounded component only; full experiments are not reproduced.
  • 2026-06 · MVEB: Massive Video Embedding Benchmark (mveb): Classification uses an 8-shot frozen logistic probe, retrieval uses cosine and nDCG@10, and VCQA uses accuracy over candidate answers. Video and audio variants are paired and only supported modalities are run. Not reproduced due to cost (review complete); research material only, no executable entry.
  • 2026-04 · NaijaS2ST: A Multi-Accent Benchmark for Speech-to-Speech Translation in Low-Resource Nigerian Languages (naijas2st): S2TT uses SSA-COMET/ChrF++; S2ST transcribes with OmnilingualASR before SSA-COMET. Text exact match is not a substitute. Not reproduced due to cost (review complete); research material only, no executable entry.

2025

2023

2021