What does multimodal alignment align: vectors, visual interfaces, or evidence dependence?¶
简体中文 · English
Updated 2026-10-29 · 20 topic entries / 6 representative comparisons
CLIP and EmbeddingGemma 2 produce comparable vectors. BLIP-2 and LLaVA connect vision to conditional generation, CV-OPD trains visual evidence use, and UMRM scores responses. Their supervision and interfaces differ, not a single capability ladder.
Mechanisms · Trade-offs · Cross-paper questions · Updates
How to read this card
The table compares representative methods. It is not a shared-benchmark leaderboard. Teaching examples are not measured results.
Mechanism comparison¶
Axes: Retrieval vectors, visual tokens, generated responses, or rewards · Dual towers, connectors, policy distributions, or heads on frozen features · Paired similarity, instruction labels, view contrasts, or perturbation preferences · Visual sequence length, embedding width, teacher, and scorer costs
| Paper and source | Mechanism | Benefits and trade-offs |
|---|---|---|
| Learning Transferable Visual Models From Natural Language Supervision · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Independent image and text encoders map to the unit sphere with symmetric in-batch contrastive loss. | Category text can be a zero-shot prototype, but similarity is not open answers. Local clothing retrieval does not reproduce 400M image-text pretraining. |
| BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Q-Former extracts visual features with a fixed set of learned cross-attention queries and feeds a frozen LLM. | The trainable bottleneck limits visual sequence length and can miss detail. Four local queries do not reproduce full two-stage training. |
| Visual Instruction Tuning · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界 |
Project vision into the language-token space and learn conditional responses with multimodal instructions. | Shape alignment is not semantic or instruction alignment. A local lightweight QA head does not generate GPT-4 instruction data or train LLaVA-13B. |
| EmbeddingGemma 2: Unified and Efficient Multimodal Representations · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 结果与边界 |
Independent modality towers produce shared 768-dimensional vectors; MRL supports specified prefix truncation followed by renormalization. | Narrower embeddings and fewer visual inputs are distinct budgets. Official local checkpoint validation covers text and images, not all four modalities. |
| CV-OPD: Contrastive Vision On-policy Distillation · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 本地结果与边界, 2026-10-14 A100 实测 |
The student reads the full image; positive and negative teachers differ only in crop and share the student prefix. JSD and CDL constrain evidence use. | Negative crops and sample/token gates shape supervision. Local evidence includes actual VLM L1 updates and A100 diagnostics, not complete visual capability. |
| Unsupervised Multimodal Reward Modeling · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 数据协议与verifier边界, 本地结果与验证状态 |
Freeze VLM hidden states, pool only the response, and train a bias-free reward head on caption perturbation preferences. | No human preferences does not mean no labels: layer selection and a bootstrap verifier use supervision. Local reward accuracy does not ensure policy-selection utility. |
Trade-off analysis¶
A shared retrieval space is not a language input space¶
CLIP and EmbeddingGemma 2 align vector similarities; the latter supports selected MRL dimensions. BLIP-2 queries and LLaVA projections condition an LLM and require generative training to use them. Matching dimensions or retrieval semantics do not make a vector a general-purpose visual soft prompt.
Sources: Learning Transferable Visual Models From Natural Language Supervision · EmbeddingGemma 2: Unified and Efficient Multimodal Representations · BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models · Visual Instruction Tuning
A good-looking answer and genuine image evidence use differ¶
UMRM learns rewards from caption perturbations and checks selection of policy-generated responses, rather than only hand-built negatives. CV-OPD changes teacher crops under a fixed student prefix to constrain distributions directly. Evaluation and distillation can complement each other without a reward benchmark proving visual attribution.
Sources: Unsupervised Multimodal Reward Modeling · CV-OPD: Contrastive Vision On-policy Distillation · Visual Instruction Tuning
October 29: publication update and evidence boundary¶
The formal EmbeddingGemma 2 paper is the October 6 model, not a new identity. It describes pooling 512-dimensional backbone features then projecting to 768. Pinned official code projects before returning last_hidden_state; equal-weight masked means commute with that linear map. Keep the 16-image GPU result at its prior evidence grade, not audio/video or seven-benchmark acceptance.
Sources: Learning Transferable Visual Models From Natural Language Supervision · EmbeddingGemma 2: Unified and Efficient Multimodal Representations
Cross-paper questions¶
Try answering before expanding the explanation. Each question relates at least two papers.
Why doesn't correct image retrieval with CLIP or EmbeddingGemma 2 show LLaVA can answer image questions?¶
Show answer and example
Answer: Retrieval matches candidate representations. LLaVA generates answer tokens from an image, question, and context, which needs a generative interface and supervision. Compare the same output task, not retrieval hits pooled with answer correctness.
Illustrative example: Retrieving an image of cups does not establish counting them, citing evidence, or following an answer format.
Sources: Learning Transferable Visual Models From Natural Language Supervision · EmbeddingGemma 2: Unified and Efficient Multimodal Representations · Visual Instruction Tuning
Do fewer BLIP-2 queries and narrower EmbeddingGemma 2 vectors save the same resource?¶
Show answer and example
Answer: Q-Former queries determine visual sequence length to the LLM. MRL width affects retrieval-vector storage and similarity. Both can lose information but at different stages; use trained prefixes with renormalization, not coordinate deletion as visual pooling.
Illustrative example: Thirty-two visual tokens and a 256-dimensional vector are length and width, not directly comparable.
Sources: BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models · EmbeddingGemma 2: Unified and Efficient Multimodal Representations
Does UMRM detecting a caption color perturbation show the CV-OPD student uses the right image region?¶
Show answer and example
Answer: UMRM ranks one negative distribution, while policy errors can differ. CV-OPD compares student and positive/negative crop teachers on a shared prefix. Correct judging, receiving that training signal, and genuinely changing evidence use are separate checks.
Illustrative example: A judge spotting red changed to blue does not show the student stopped guessing common colors.
Sources: Unsupervised Multimodal Reward Modeling · CV-OPD: Contrastive Vision On-policy Distillation
When can prior results survive a difference between paper and code operator order?¶
Show answer and example
Answer: Check the actual operations. The same bias-free linear projection commutes with equal-weight masked averaging and the pinned code/checkpoint did not change. Nonlinearity, different weights, or retraining do not inherit equivalence. Only provenance changes here.
Illustrative example: mean(W h1,W h2)=W mean(h1,h2); averaging two GELU outputs generally does not equal GELU of their mean.
Sources: Learning Transferable Visual Models From Natural Language Supervision · EmbeddingGemma 2: Unified and Efficient Multimodal Representations
Update log¶
Updates record new approaches, changed trade-offs, and follow-up evidence.
2026-10-29 · The formal EmbeddingGemma 2 paper describes the October 6 model, so no duplicate identity is added. It pools 512-dimensional features before projection to 768; pinned code projects before last_hidden_state, equivalent under equal-weight masked averaging. The prior 16-image GPU run does not qualify audio, video, or seven full benchmarks.
Affected: EmbeddingGemma 2: Unified and Efficient Multimodal Representations
2026-10-27 · Initial version: compared CLIP and EmbeddingGemma 2 retrieval, BLIP-2/LLaVA interfaces, CV-OPD evidence use, and UMRM rewards; screened 20 abstracts and read six papers closely.
Covers 20 topic entries; see the full list below.
All 20 topic entries
The full catalog follows. The main comparison covers the representative papers listed above.
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Learning Transferable Visual Models From Natural Language Supervision
- From Visual Cues to Spoken Narration: Rethinking Audio Description
- CV-OPD: Contrastive Vision On-policy Distillation
- EmbeddingGemma 2: Unified and Efficient Multimodal Representations
- FlexGlue: Sparse-to-Dense Feature Matching and Visual Disambiguation
- Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction
- LAMP: Layerwise Adaptive Mixing of Predictors for Training-Free Few-Shot Vision-Language Adaptation
- Visual Instruction Tuning
- MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations
- MVEB: Massive Video Embedding Benchmark
- NaijaS2ST: A Multi-Accent Benchmark for Speech-to-Speech Translation in Low-Resource Nigerian Languages
- Omni-Embed-Mini: Binding Modalities Without Forgetting via Dense Distillation
- Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
- ReToken: One Token to Improve Vision–Language Models for Visual Retrieval
- SenseNova-U1.5: Towards Native Unified Visual Intelligence
- SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
- SmolVLM: Redefining small and efficient multimodal models
- Unsupervised Multimodal Reward Modeling
- WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report