跳转至

Multimodal models by institution, company, or university

简体中文 · English

Grouped by the first listed affiliation of the first author, newest first within each group.

Aalto University

  • 2026-10-20 · Multimodal Reranking Is a System One Decision (system-one-rerank): Instead of generating a ranked list or training a separate pointwise reranker, the model scores each candidate by reading the probability of its letter label at the first answer position. The readout follows setwise and FIRST-style methods; the paper tests how this approach works zero-shot on images and visual documents, and whether text-only decision tuning transfers to visual reranking. Bounded component only; full experiments are not reproduced.

Alibaba Cloud Computing

ByteDance

City University of Hong Kong

ETH Zurich

  • 2026-10-23 · FlexGlue: Sparse-to-Dense Feature Matching and Visual Disambiguation (flexglue): It reads a local 3×3 neighborhood to avoid averaging distant modes, then refines subpixel locations with expected offsets from two axis-specific bins. A per-layer sigmoid co-visibility vote supports early rejection, trained with non-overlapping and flip-derived hard negatives rather than a gold-aware filter. Not reproduced due to cost (review complete); research material only, no executable entry.

Google

  • 2026-10-06 · EmbeddingGemma 2: Unified and Efficient Multimodal Representations (embedding-gemma2): The technical paper reports 744M total parameters: 271M in the text backbone including the embedder, 167M in vision, and 305M in audio. The authors give 439M for the image–text path; these are rounded approximations. Checkpoint inference; not a reproduction of training.

Google DeepMind

Harvard University

  • 2026-06-12 · MVEB: Massive Video Embedding Benchmark (mveb): Classification uses an 8-shot frozen logistic probe, retrieval uses cosine and nDCG@10, and VCQA uses accuracy over candidate answers. Video and audio variants are paired and only supported modalities are run. Not reproduced due to cost (review complete); research material only, no executable entry.

Hugging Face

  • 2025-04-07 · SmolVLM: Redefining small and efficient multimodal models (smolvlm): Move neighboring spatial tokens into channels with pixel shuffle, then project into LM space with an MLP. The paper also studies long contexts, image tiling, learned position tokens and data mixtures for 256M/500M/2.2B image/video models.

Massachusetts Institute of Technology

  • 2026-10-08 · Unsupervised Multimodal Reward Modeling (umrm): A second experiment generates the policy's own candidates, creates success/failure pairs with a verifier, and retrains the head. Deployment utility improved while reward-benchmark scores fell.

Mila - Quebec AI Institute

Northwestern University

  • 2026-10-08 · WOVEN: Weaving Visual World Modeling into Multimodal LLMs (woven): The mechanism is training a multimodal LM with actual visual-transition supervision, not an image scorer. The corpus contains 36,076 four-option questions, 20 scene types, 5 action types, and 8 operations: 22,728 train, 2,508 validation, 6,228 ID test, 3,496 scene-OOD, and 1,116 state-perturbation items.

OpenAI

Salesforce Research

Sichuan University

Stanford University

  • 2026-10-23 · UniAction: Unified Action Conditioning for World Models (uniaction): UniAction represents actions as sparse elements with discrete semantic IDs and continuous time/space coordinates, looks up a shared codebook and delegates position encoding to the video backbone. A group-wide log bias corrects action mass without changing the relative action logits. Bounded component only; full experiments are not reproduced.

TU Darmstadt & hessian.AI

  • 2026-09-01 · From Visual Cues to Spoken Narration: Rethinking Audio Description (cue2narrate-drl): DRL applies negative log-sigmoid to the score difference divided by 0.5 and enters the full loss with weight 0.2. The offline negative-caption generator is frozen, but the current model's likelihood of the generated caption still backpropagates. Bounded component only; full experiments are not reproduced.

UC Berkeley

  • 2026-10-21 · Agentic World Action Models (aw0): Previous fragments become learned memory. Code normally conditions control without execution; optional hybrid executes skills only after validity, precondition and progress checks, otherwise falls back to direct actions. Not reproduced due to cost (review complete); research material only, no executable entry.

University of Modena and Reggio Emilia

  • 2026-06-30 · Editing Everything Everywhere All at Once (mice): MICE isolates text groups in joint attention and uses a smooth spatial bias from instance masks to control visual interactions. This balances local binding and blending with the surrounding region without training a new editing model. Bounded component only; full experiments are not reproduced.

University of Science and Technology of China

University of Wisconsin-Madison

  • 2023-04-17 · Visual Instruction Tuning (llava): Freeze a vision encoder and project its features to the LLM token space, then tune with GPT-4-generated multimodal instructions.

Xiamen University

  • 2026-10-09 · CV-OPD: Contrastive Vision On-policy Distillation (cv-opd): Three constraints matter: the positive and negative teachers differ only in their crop and see the same student prefix; CDL uses a bounded-JSD margin; and DAG uses a strict greater-than-mean token threshold over the entire retained batch. A sample is retained only if at least one rollout is correct, and then all its rollouts are retained.