Cross-modal alignment: representations, connectors, and supervision¶
简体中文 · English
Multimodal models / 81 questions · 20 papers
Try answering each question before opening the explanation. The bank grows as papers and questions are added.
Read this topic's comparisons, analysis, and cross-paper Q&A
81 questions · 20 papers
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
Are Q-Former queries the text of the question to answer?
Show answer and example
Answer: They are a fixed number of learnable vectors, not the user's question. Cross-attention reads tokens from the frozen vision encoder and passes selected information to the language side. Their fixed count forms a visual bottleneck: more complex images do not automatically produce more output tokens.
Illustrative example: Four learnable queries can attend to hundreds of image patches while sending only four vectors to the language model.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
How can Q-Former learn to connect images to a language model when both the vision encoder and language model are frozen?
Show answer and example
Answer: Gradients can still flow through frozen modules to the connector; only parameter updates are frozen. Learnable queries select visual information through cross-attention and are projected into soft prompts the language model can consume. Two training stages address vision–language representation alignment and the generation interface. The bottleneck can still lose detail.
Illustrative example: Compressing 16 image features into four queries gives the language model four visual prompts, but reading small text may need more capacity.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
How does BLIP-2's interface differ from linearly projecting each visual patch?
Show answer and example
Answer: Per-patch projection primarily changes feature dimensions and usually preserves token count. Q-Former actively aggregates visual information with learnable queries before projection. This changes both selection and sequence length; lost detail depends on training and the task, not architecture alone.
Illustrative example: Clothing recognition may need only an outline, while reading small text may need local detail. Four queries are not guaranteed to suffice for both.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
Do the local four-query results validate the paper's two-stage pretraining?
Show answer and example
Answer: No. The local work uses real Fashion-MNIST pixels and four-query cross-attention, without ViT-g, a large LM, ITC/ITM/ITG, or large-scale second-stage training. It checks token compression and small-budget behavior, not original zero-shot capability.
Illustrative example: Reducing 16 visual tokens to four proves this interface shortens the sequence, not that it pretrained on over 100 million images.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
Why can CLIP do zero-shot classification without retraining a classifier head?
Learning Transferable Visual Models From Natural Language Supervision
Show answer and example
Answer: Separate encoders normalize image and class-description vectors and compare them in a shared space. The descriptions are class prototypes and can be replaced to define new labels. This depends on pretraining and does not guarantee arbitrary new concepts or fine-grained relationships are recognized.
Illustrative example: Compare a shoe image with descriptions such as “a shoe” and “a coat,” then choose the higher similarity without training another linear head.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
Why does CLIP use both image-to-text and text-to-image contrastive learning?
Learning Transferable Visual Models From Natural Language Supervision
Show answer and example
Answer: The same batch supplies two classification problems: find each image's paired text and each text's paired image. Both directions shape the shared similarity space. Other in-batch pairs provide negatives, but overall matching scores do not establish fine-grained factual accuracy.
Illustrative example: Cat and dog images pair with their respective descriptions, and both retrieval directions should rank each matching pair first.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
How do CLIP's class text embeddings differ from autoregressive label generation?
Learning Transferable Visual Models From Natural Language Supervision
Show answer and example
Answer: CLIP compares similarities among supplied class texts and returns a choice or ranking within the candidates. An autoregressive model predicts a label token sequence that requires generation and parsing. CLIP's descriptions are input prototypes, not generated targets, so this interface does not provide open-ended captioning.
Illustrative example: Given “a photo of a cat” and “a photo of a dog,” CLIP compares their scores without generating “This is a cat.”
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
Why normalize each vector before computing CLIP's similarity matrix?
Learning Transferable Visual Models From Natural Language Supervision
Show answer and example
Answer: Normalization makes dot products track direction instead of letting vector norms inflate scores; temperature controls sharpness. Matching pairs are on the diagonal, and the two directional losses use the matrix and its transpose. Not every pairing is a positive.
Illustrative example: Doubling an image vector should not make it more correct for every text. With unit normalization only a direction change alters cosine similarity.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
Why pair the negative caption with the same visual cue and prompt?
From Visual Cues to Spoken Narration: Rethinking Audio Description
Show answer and example
Answer: Matching conditions isolates whether the descriptions suit the intended audio-description task. Changing image or prompt could instead change task difficulty. Offline negative captions are training material, not an extra deployment generation step.
Illustrative example: Score “She sets down the cup” and an unrelated-action caption with the same film clip and instruction.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
Why backpropagate through a negative caption's current likelihood if a frozen model generated it?
From Visual Cues to Spoken Narration: Rethinking Audio Description
Show answer and example
Answer: Freezing its generator fixes the negative text, not the trainable model's scores. DRL compares mean target-token log likelihoods for positive and negative texts with the same visual input. Gradients improve their relative preference; detaching the negative score changes the contrastive optimization.
Illustrative example: For the same scene, one caption says a person walks and the other says they run. The strings are fixed, but the trainable model still changes their relative scores.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
What does DRL add beyond SFT on positive descriptions?
From Visual Cues to Spoken Narration: Rethinking Audio Description
Show answer and example
Answer: SFT raises correct target-token likelihood; DRL explicitly separates positive and negative mean log probabilities. This adds a relative signal whose meaning depends on how negatives are built. Trivial negatives do not automatically teach detailed visual grounding.
Illustrative example: Raising the likelihood of “She sits down” need not lower the model's score for the wrong “She runs away.” The relative loss constrains the pair directly.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
Can SFT and DRL use one shared length normalization?
From Visual Cues to Spoken Narration: Rethinking Audio Description
Show answer and example
Answer: No. The pinned SFT sums valid target tokens per sample before the batch mean. DRL averages token log probabilities separately in each description. Both exclude prompt and padding; freezing the negative generator does not stop gradients through the current model's negative-caption scores.
Illustrative example: For 10 positive tokens and 20 negative tokens, DRL compares their means. Raw sums would additionally penalize the longer description.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
Why prepare positive and negative crops from the same image in CV-OPD?
CV-OPD: Contrastive Vision On-policy Distillation
Show answer and example
Answer: The positive provides local evidence; the same-image, same-size, low-overlap negative is a visual counterfactual. The student sees the full image, while teacher inputs differ only in region. This tests whether learning follows the evidence rather than a language prior.
Illustrative example: For a street sign, use the sign in the positive crop and nearby wall in the negative crop. Both teachers share parameters.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
Why does CV-OPD give both teachers the same student prefix and change only the visual crop?
CV-OPD: Contrastive Vision On-policy Distillation
Show answer and example
Answer: Differences then primarily come from the region, not text history. On its own rollout prefixes, the student moves toward the correct-region teacher and away from the negative-region teacher by a bounded JSD margin. This addresses answer distillation that might learn only a language prior.
Illustrative example: For cup color, the positive crop shows the cup and the same-size negative crop shows background. Both teachers see identical student-generated text so far.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
What does CDL add over distillation from a positive-crop teacher alone?
CV-OPD: Contrastive Vision On-policy Distillation
Show answer and example
Answer: Positive-only distillation pulls the student toward the positive distribution. CDL also requires sufficient JSD from the negative teacher using a margin on a bounded divergence. It neither requires unlimited separation nor guarantees improvements on every benchmark.
Illustrative example: A student equally close to both teachers may have low positive-only loss, but still incur a margin penalty for failing to distinguish regions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
If at least one rollout is correct, should the other rollouts be dropped?
CV-OPD: Contrastive Vision On-policy Distillation
Show answer and example
Answer: No. Keep all rollouts of a sample that passes the at-least-one-correct gate. DAG gates tokens across the retained batch with JSD strictly above the mean. Teachers, generation, and gate do not backpropagate. The top-100 JSD approximation must preserve the full tail bucket.
Illustrative example: If one of eight rollouts is correct, keep all eight; removing the other seven changes the training distribution.
Sections in the Chinese source: 核心公式, 本地复现, 本地结果与边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
Does a shared 768-dimensional space require all four modalities at once?
EmbeddingGemma 2: Unified and Efficient Multimodal Representations
Show answer and example
Answer: No. A shared space lets all modalities use comparable vector similarities, while towers load as needed. Supported inputs and actually verified paths differ. Local official-checkpoint validation covers text and images only, not audio and video pipelines.
Illustrative example: Image-to-text search needs only the image and text paths; their validation does not establish audio retrieval.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
Why renormalize a truncated Matryoshka embedding?
EmbeddingGemma 2: Unified and Efficient Multimodal Representations
Show answer and example
Answer: The retained prefix generally no longer has unit norm. Raw dot products would mix differences in retained energy with similarity. Renormalizing restores cosine scale. This relies on trained prefix representations, not arbitrary truncation, and pooling must exclude padding.
Illustrative example: Truncate two 768-dimensional vectors to 128 dimensions and normalize each again so larger retained norms do not confer an advantage.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
How does MRL truncation differ from randomly dropping dimensions?
EmbeddingGemma 2: Unified and Efficient Multimodal Representations
Show answer and example
Answer: MRL trains designated prefixes, such as 128, 256, or 512 dimensions, to serve as representations after renormalization. Random subsets lack that training convention. Shorter prefixes save storage and retrieval cost, but quality must be reported by task and fixed size.
Illustrative example: Taking the first 128 of 768 dimensions follows the trained contract; randomly drawing 128 each time is a different compression method.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
The formal paper pools before projection, but the local path pools official last_hidden_state. Is a layer missing?
EmbeddingGemma 2: Unified and Efficient Multimodal Representations
Show answer and example
Answer: No. The pinned official implementation applies a bias-free embedding_projection before returning its 768-dimensional token states. Equal-weight masked mean commutes with linear projection, so pooling those outputs matches the paper; projecting again would be incorrect. The formal paper does not upgrade the existing 16-image GPU run to seven full benchmarks.
Illustrative example: Averaging two tokens after multiplying each by the same matrix equals averaging first and then multiplying. Nonlinearity or token-dependent transformations break this equivalence.
Sections in the Chinese source: 核心公式, 结果与边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
What details matter for mean pooling, truncation, and precision?
EmbeddingGemma 2: Unified and Efficient Multimodal Representations
Show answer and example
Answer: Attention-mask pooling excludes padding, followed by prefix truncation and L2 renormalization. Queries and documents use their respective text prefixes; images use none. Official activation ranges can produce non-finite FP16 values, so the local path pins FP32 and rejects non-finite or zero vectors.
Illustrative example: Identical text padded to different lengths should yield the same masked representation. Including padding makes it length-dependent.
Sections in the Chinese source: 核心公式, 本地复现, 结果与边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
How does FlexGlue's dense grid support arbitrary-position queries?
FlexGlue: Sparse-to-Dense Feature Matching and Visual Disambiguation
Show answer and example
Answer: Features interact globally on a stride-8 grid. Interpolation reads the requested coordinate's feature, and cross-image attention estimates the corresponding location. Queries need not coincide with cell centers or be restricted to fixed keypoints.
Illustrative example: For a query between four cells, interpolate its feature before matching instead of snapping to the nearest cell.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
Why does FlexGlue read coordinates near a peak rather than take a global expectation?
FlexGlue: Sparse-to-Dense Feature Matching and Visual Disambiguation
Show answer and example
Answer: Similar structures can create several peaks, making a global average land between valid matches. A local window around the strongest candidate avoids averaging modes; refinement then predicts a displacement. Covisibility votes and hard negatives separately assess whether the images are matchable.
Illustrative example: If two identical windows score highly, the global average lands on the wall. Read coordinates around the chosen window for a plausible starting point.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
When does FlexGlue's layer-wise covisibility gate exit early?
FlexGlue: Sparse-to-Dense Feature Matching and Visual Disambiguation
Show answer and example
Answer: Every layer produces token-level sigmoid votes. Exit when the mean across both images falls below that layer's threshold tγ^(L−k). This tests whether matching should continue, not just the strongest local match. Reliability of the gate matters; fewer layers do not prove higher quality.
Illustrative example: Two symmetric buildings can contain nearly identical local features but have low overall covisibility, so exit rather than refine a confident false match.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
Can ordinary softmax and cross-entropy reproduce FlexGlue's coarse matching?
FlexGlue: Sparse-to-Dense Feature Matching and Visual Disambiguation
Show answer and example
Answer: No. The paper uses 1.5-entmax and Fenchel–Young loss, with bilinear weights on four cells surrounding the subpixel target and detached coarse coordinates during refinement. No full local operator, training metrics, or GPU receipt exists.
Illustrative example: A target at a four-cell junction gives all four weights. Snapping to one cell changes the supervision.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
Does GAS next-embedding prediction output discrete image tokens?
Show answer and example
Answer: No. It predicts continuous embeddings in the LM's visual input space, with cosine alignment to an EMA-projected target. It does not classify a discrete image codebook. Generation is auxiliary to understanding and deployment need not generate images.
Illustrative example: Predict the direction of a target patch embedding instead of choosing an index from thousands of image codes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
Why can GAS remove its entire auxiliary generation branch at deployment?
Show answer and example
Answer: During training it predicts continuous visual embeddings and backpropagates into a lower vision path shared with understanding. Its upper generation parameters are separate and not used by inference. Removing the branch retains the trained shared parameters.
Illustrative example: Train on image understanding and target visual embeddings, then deploy only image question answering with the learned shared vision path.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
How does GAS next-embedding prediction differ from reconstructing the current patch?
Show answer and example
Answer: NEP predicts the next position from prior visual embeddings and allowed autoregressive context; an EMA projection supplies the target. Feeding the target at the same position changes what the model sees and can reduce the task to copying. Removability of the branch does not replace the shifted objective.
Illustrative example: To predict visual embedding i, provide earlier positions and allowed context, not embedding i itself.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
Why check inference dependencies before claiming GAS adds no deployment parameters?
Show answer and example
Answer: The generation upper layers, vision head, and EMA projector are removable only if understanding no longer reads them. Training still needs shifted prediction, EMA targets, and branch isolation. A local mirror-target diagnostic cannot validate the gains of large-scale generative data.
Illustrative example: Feeding an intermediate generation output back into understanding keeps that branch necessary at inference despite a claim of zero added parameters.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
Relative to what does LAMP compute its label residuals?
LAMP: Layerwise Adaptive Mixing of Predictors for Training-Free Few-Shot Vision-Language Adaptation
Show answer and example
Answer: Subtract the original CLIP text-prototype scores from true support labels. Each layer's kernel predictor corrects the prior rather than relearning all classes. If prototypes change, the residual must follow or the correction no longer matches its target.
Illustrative example: If a cat support image has a low CLIP cat score, the kernel branch learns the difference rather than retraining the vision encoder.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
Why mix LAMP's layer predictions in class-score space rather than concatenate features?
LAMP: Layerwise Adaptive Mixing of Predictors for Training-Free Few-Shot Vision-Language Adaptation
Show answer and example
Answer: Different layers have different widths and similarity geometry. Concatenation would introduce projection assumptions. Each LAMP branch fits support-set residuals relative to CLIP, then mixes scores in common class space. If text prototypes change, the residuals must change as well.
Illustrative example: A shallow texture branch and a deeper semantic branch each score cat versus dog before their scores are combined.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
How does a support-bank regional kernel differ from including evaluation queries in its centering statistics?
LAMP: Layerwise Adaptive Mixing of Predictors for Training-Free Few-Shot Vision-Language Adaptation
Show answer and example
Answer: Both m(x) and b use the support set as the reference; queries are scored against the fixed reference. Adding test queries would make an individual result depend on the evaluation batch. Test labels must not select mixing weights.
Illustrative example: With fixed support, query A uses the same calibration alone or with query B; B does not redefine the support mean.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
Why must LAMP-F update the cached response when text prototypes change?
LAMP: Layerwise Adaptive Mixing of Predictors for Training-Free Few-Shot Vision-Language Adaptation
Show answer and example
Answer: Updating prototypes changes both query priors and support residuals. The cached linear response represents how the kernel solution changes and allows exact updates without a complete solve. Updating just the query prior omits a term. Kernel statistics and mixture weights remain frozen.
Illustrative example: Moving the prototype toward cat images raises the prior for support cats, so their residual should shrink. Keeping the old correction double counts.
Sections in the Chinese source: 核心公式, 本地复现, 代码映射与边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
What does LLaVA's visual projector do between image and answer?
Show answer and example
Answer: It maps image-encoder features into visual tokens that the language model reads alongside questions and conversation history. It connects the interfaces; multimodal instruction training, not shape alone, teaches instruction following.
Illustrative example: A width-matching projection maps encoder features to LM size, but the model still must learn which features address the user's question.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
Why is projecting visual features to the LLM width insufficient for following visual instructions?
Show answer and example
Answer: Matching shape does not align meaning. The connector learns a visual-to-language interface, and instruction supervision teaches how to answer questions conditioned on images. Projection alone does not confer recognition, localization, or multiround responses.
Illustrative example: A cat image can enter as a vector, but QA training is needed to answer “Where is it?”
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
How does LLaVA instruction supervision differ from CLIP's dual-encoder contrastive training?
Show answer and example
Answer: LLaVA learns conditional answer tokens from images and questions. CLIP brings paired images and texts together in a shared embedding space for matching. Retrieval scores are not instruction-generation quality, and architecture alone does not establish superiority.
Illustrative example: Answering why two people hold umbrellas is conditional generation; selecting a supplied matching caption is a different output contract.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
Does the local two-layer projector's negative result refute LLaVA's instruction tuning?
Show answer and example
Answer: No. The local run fixes a small patch encoder, classification head, and budget, and compares a linear projector with a two-layer MLP. It does not generate GPT-4 instructions or train LLaVA-13B. Extra connector capacity did not help this configuration, not all tasks.
Illustrative example: A decline in small clothing classification says nothing directly about large-model multiround image dialogue.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
Why use a multimodal LLM teacher but keep a dual-tower MLLMCLIP student?
MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations
Show answer and example
Answer: The teacher supplies rich cross-modal semantics but is expensive to deploy as a generator. The student distills relationships while encoding images and text independently. Teacher training cost and online retrieval cost are separate.
Illustrative example: Distill offline, cache item image embeddings, and retrieve with a text vector without generating an answer for each item.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
Why can CKA distill structure when teacher and student feature widths differ?
MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations
Show answer and example
Answer: CKA aligns relationships between examples without coordinate-by-coordinate equality. The teacher selects informative tokens by attention; CKA constrains student image and text representations. The deployed student stays dual-tower and does not reproduce the teacher's entire reasoning.
Illustrative example: A 1,000-dimensional teacher and 500-dimensional student can both put similar examples close without matching coordinates.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
How does attention token selection change MLLMCLIP supervision relative to distilling all tokens?
MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations
Show answer and example
Answer: It builds structural supervision from selected informative teacher positions across layers rather than every position. This changes information selection, not CKA's ability to handle different widths. Selection can miss useful details, so superiority requires a controlled comparison.
Illustrative example: Select a subset of locations when a teacher layer includes many background tokens. More tokens do not necessarily give the student more useful knowledge.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
Does the real-checkpoint path train the full CLIP student?
MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations
Show answer and example
Answer: No. It freezes SmolVLM2 and the CLIP encoders, trains a dimension-matching feature projection, and selects weights on separate validation. Held-out CKA and neighbor overlap do not improve consistently across three seeds. A perfect score caused by degenerate labels was invalidated.
Illustrative example: Running CKA after projection checks the operator path, not full student training over the paper's 26 benchmarks.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
Why is MVEB's video question answering still an embedding evaluation?
MVEB: Massive Video Embedding Benchmark
Show answer and example
Answer: VCQA embeds candidate answers in a retrieval pool, chooses a match for the video, and reports pool accuracy. It is not free generation. The embedding model is frozen; classification and other tasks use their prescribed probes, so output metrics differ by task.
Illustrative example: Select the closest of four supplied action descriptions rather than freely generating an action and judging its wording.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
Why use identical clips for MVEB's video-only and audio-plus-video comparison?
MVEB: Massive Video Embedding Benchmark
Show answer and example
Answer: Otherwise changes in sample difficulty confound the contribution of audio. Keep clips paired, encoders frozen, and task-specific probes or retrieval metrics fixed; respect each model's frame and audio configuration. An invented audio score for an unsupported model invalidates the comparison.
Illustrative example: Evaluate the same door-knocking video with and without sound, not a different easier clip.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
Is forcing every MVEB video model to take 64 frames fairer than respecting its configuration?
MVEB: Massive Video Embedding Benchmark
Show answer and example
Answer: Not necessarily. MVEB uses 2 fps and up to 64 frames for variable-length models, but follows training configuration for fixed-frame models. Fixed clips are needed for audio/video pairing, and models without audio support must not receive fabricated results.
Illustrative example: Feeding 64 frames into an eight-frame model and calling the failure low embedding quality is not a fair test.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
Can a single cosine aggregator implement all MVEB tasks locally?
MVEB: Massive Video Embedding Benchmark
Show answer and example
Answer: No. Official task evaluators include retrieval nDCG@10, clustering V-measure, eight-example-per-class logistic regression, and candidate QA accuracy, each with input and sampling constraints. The local page reviews resources and cost only; no model calls or formal metrics.
Illustrative example: Retrieval must calculate nDCG from relevance labels. Mean cosine is neither a replacement nor QA accuracy.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
What does accent diversity require in NaijaS2ST evaluation?
Show answer and example
Answer: Pronunciation variation affects understanding and speech evaluation. Cover multiple speakers and accents, report translation directions separately, and distinguish speech-to-text from speech-to-speech. One direction or a few speakers cannot characterize all Nigerian languages or accents.
Illustrative example: English→Hausa is not Hausa→English, and understanding one accent does not prove reliable synthesis of another.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
Why not use exact match against reference text for speech-to-speech translation?
Show answer and example
Answer: Audio output needs real ASR before scoring with prescribed semantic translation metrics. Both transcription and translation errors matter. Exact text matching rejects valid paraphrases and bypasses audio. Few-shot examples come from development data, not test references.
Illustrative example: Two spoken versions of “Please come tomorrow” can both be correct with different wording; preserve both the semantic and audio pipeline.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
Do SSA-COMET and ChrF++ measure the same kind of agreement?
Show answer and example
Answer: SSA-COMET uses a multilingual semantic model; ChrF++ mainly uses character and word n-gram overlap. The scores are complementary and not interchangeable accuracy percentages. Speech output still needs the prescribed ASR path.
Illustrative example: Two valid paraphrases can preserve meaning with different character overlap; the larger number is not automatically a larger fraction correct.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
What limits apply to NaijaS2ST five-shot examples and proxy language tokens?
Show answer and example
Answer: Take five examples from DEV, never formal test. Some Seamless adaptations map Hausa to Arabic and Naijá to French tokens as conventions, not native model support. Local work has not downloaded audio or run complete training and evaluation.
Illustrative example: Assigning an Arabic control token for Hausa fine-tuning does not mean the audio is translated to Arabic or that untrained Hausa support exists.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
Does Omni-Embed-Mini output new media or a retrieval vector?
Omni-Embed-Mini: Binding Modalities Without Forgetting via Dense Distillation
Show answer and example
Answer: It learns a semantic media embedding comparable to description embeddings, not image, audio, or video generation. Dense descriptions supply training targets, but the media path must still encode the input itself.
Illustrative example: An audio vector comparable to a text query is representation learning; synthesizing sound from text is a generation task.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
Why can distilling from descriptions reduce disruption to the text backbone when adding media?
Omni-Embed-Mini: Binding Modalities Without Forgetting via Dense Distillation
Show answer and example
Answer: A frozen text backbone supplies stable semantic coordinates. Dense descriptions become teacher targets; a projector and controlled LoRA learn media mappings into that space. Stop gradients through the targets, and check for relevant media information that descriptions omit.
Illustrative example: An action-and-scene description anchors a video embedding, but missing small text cannot be recovered from the target alone.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
What is the trade-off between frozen-text distillation and joint full-parameter multimodal training?
Omni-Embed-Mini: Binding Modalities Without Forgetting via Dense Distillation
Show answer and example
Answer: Freezing offers stable coordinates for new modalities and limits changes to existing text representations. Full training has broader flexibility but can shift old text capabilities. The scope of updates alone does not determine which performs better.
Illustrative example: Preserving existing text vectors makes an audio retrieval index easier to maintain, but text-space constraints can still limit audio-specific distinctions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
Where should stop-gradient apply in cosine distillation?
Omni-Embed-Mini: Binding Modalities Without Forgetting via Dense Distillation
Show answer and example
Answer: Detach the description target from the frozen text backbone. Cosine loss updates only allowed media-side parameters. The local same-space dense-description diagnostic does not train the full six-modality encoder or reproduce retrieval leaderboards.
Illustrative example: Reduce the angle by moving the media mapping only. Updating both vectors would move the supposedly fixed semantic anchor.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
Does modality synergy simply mean adding more data?
Show answer and example
Answer: It asks whether training on one modality improves knowledge or representations for another. Controlled experiments distinguish synergy from competition. Adding data alone changes compute, coverage, and task weighting together.
Illustrative example: If text reasoning improves after adding image training, check total tokens and compute before attributing it to shared knowledge.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
What do shared attention and normalization versus modality-specific FFNs control?
Show answer and example
Answer: Shared components allow cross-modal interactions, while separate FFNs preserve modality-specific transformations and capacity and may reduce competition. Hold data, loss weights, and budgets fixed to identify synergy; a better aggregate does not imply equal gains for every modality.
Illustrative example: Images and text use the same attention path but separate FFNs. Evaluate each separately.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
How does early joint training differ from concatenating separately trained modality vectors?
Show answer and example
Answer: Joint losses update shared interaction parameters and can produce cross-modal gradient synergy or competition. Concatenation only changes downstream visible features after training; it does not create shared pretraining, even at equal data volume.
Illustrative example: Updating shared attention from text and retesting vision can reveal transfer. Placing two frozen vectors side by side does not create that update.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
Why are joint objective weights part of the reproduction contract?
Show answer and example
Answer: Text, understanding, and generation terms have different gradients into shared parameters. The two coefficients alter their balance and therefore the conditions for synergy or competition. A runnable mini-suite or one perfect seed does not reproduce a 13.5B MoE trained on two trillion tokens.
Illustrative example: Increasing generation weight tenfold changes optimization pressure although loss names stay the same.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
Is ReToken's learned RET token a fixed ID for one image?
ReToken: One Token to Improve Vision–Language Models for Visual Retrieval
Show answer and example
Answer: No. It is a shared learned input marker whose contextual VLM state is projected and matched to each image's average value. Shared parameters do not imply identical query states or selected images, and the token does not encode an entire catalog.
Illustrative example: Two questions with the same RET marker can yield different relevance scores for the same image pool.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
Why score retrieval in the final-layer value space in ReToken?
ReToken: One Token to Improve Vision–Language Models for Visual Retrieval
Show answer and example
Answer: Prefilled vision KV already contains image representations. Similarities in the same value space can reuse cache without rerunning an external retriever and re-encoding images. A learned retrieval token and projection adapt relevance; a frozen VLM does not eliminate supervision or guarantee generalization.
Illustrative example: With 10 prefetched images, score their average value vectors against the new query's retrieval token and reuse selected images.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
How does ReToken per-image BCE differ from softmax with one positive candidate?
ReToken: One Token to Improve Vision–Language Models for Visual Retrieval
Show answer and example
Answer: BCE scores each relevance label independently, weighting classes without forcing sigmoid scores to sum to one. Single-positive softmax makes candidates share probability mass. Do not silently discard valid additional positives while claiming the same supervision.
Illustrative example: A question can require two relevant images. Both can be positive rather than relabeling one negative to force one winner.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
What is missing between ReToken's local text cache experiment and the visual training?
ReToken: One Token to Improve Vision–Language Models for Visual Retrieval
Show answer and example
Answer: WikiText-2 trains a single retrieval target, value scoring, and sparse selection, not the frozen large VLM, MIRAGE vision-language supervision, or long video. Short prefixes cannot fill top-K and affect retention. Report worse PPL as a text mechanism result, not a visual gain.
Illustrative example: With three history tokens and top-K ten, at most three can be retained; average retention across lengths is not a fixed compression ratio.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
Do visual understanding and generation in SenseNova-U1.5 have the same input–output direction?
SenseNova-U1.5: Towards Native Unified Visual Intelligence
Show answer and example
Answer: No. Understanding conditions on vision to answer or judge, while generation produces visual content from conditions. A unified model needs both signals. Spatial patch reconstruction concerns generation representations; expert OPD concerns post-training supervision. One operator does not establish both capabilities.
Illustrative example: Answering what is in an image and creating an image on request are different acceptance tests.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
How does spatial patch reconstruction differ from predicting discrete image codes?
SenseNova-U1.5: Towards Native Unified Visual Intelligence
Show answer and example
Answer: Discrete methods use a tokenizer or VAE to turn images into codes, then predict those codes. Patch reconstruction targets local spatial structure directly and relies less on that intermediate interface. This changes representation and supervision, not proof of zero bottleneck or lower cost.
Illustrative example: One method predicts codebook IDs and the other reconstructs spatial patches. Their error sources differ.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
How does multi-expert OPD differ from a single teacher?
SenseNova-U1.5: Towards Native Unified Visual Intelligence
Show answer and example
Answer: Experts supply different tasks or abilities; the unified model must reconcile signals rather than match one teacher distribution. Coverage can grow, but so can conflicts. More teachers do not guarantee improvement in both understanding and generation.
Illustrative example: A teacher emphasizing object relations and another emphasizing image reconstruction do not simply repeat the same target.
Sections in the Chinese source: 背景与主要改动, 本地复现. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
Do the local masked-patch and expert OPD results establish a unified 8B model?
SenseNova-U1.5: Towards Native Unified Visual Intelligence
Show answer and example
Answer: No. Deterministic reference kernels and three-seed state-transition diagnostics cover masked targets and expert fusion, not multistage 8B training. Gating and loss values verify paths, not real understanding or generation benchmarks.
Illustrative example: Correctly summing expert weights in a tensor does not demonstrate usable images or real visual answers from an eight-billion-parameter model.
Sections in the Chinese source: 核心公式或操作, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
Does SigLIP 2 NaFlex change the number of class labels?
Show answer and example
Answer: No. Dynamic resolution changes how vision input is represented and where compute goes. Labels are separate. SigLIP 2 combines input adaptation, global semantics, and local feature training; a fixed-resolution loss kernel does not cover the complete recipe.
Illustrative example: The same cat and dog labels can accompany images of different sizes.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
Why add local self-distillation and masked prediction to pairwise sigmoid alignment?
Show answer and example
Answer: Global matching constrains whole-image compatibility with text but need not train local or dense features. Local distillation and masking supervise spatial detail; captioning and data selection improve semantic inputs. Ablate these separately rather than credit sigmoid alone.
Illustrative example: A street caption may only mention a car, while recovery around its masked wheel encourages finer spatial features.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
How does pairwise sigmoid differ from CLIP batch softmax?
Show answer and example
Answer: Softmax normalizes over each row or column so a matching pair competes with other batch entries. Sigmoid gives each image–text pair an independent positive or negative classification term. Normalization, temperature, and bias differ; effectiveness is empirical.
Illustrative example: Cat image–cat text is positive and cat image–dog text negative; sigmoid applies two separate binary constraints.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
How do label signs and bias enter sigmoid pair loss?
Show answer and example
Answer: Use z=+1 or −1 in log(1+exp(−z·score)), with scaled dot product plus bias as score. Substituting 0/1 without changing the formula removes the negative gradient. Local work covers class text and masked-view consistency only.
Illustrative example: z=−1 penalizes high negative-pair similarity; z=0 makes loss constant.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
Does SmolVLM just remove language-model layers?
SmolVLM: Redefining small and efficient multimodal models
Show answer and example
Answer: No. It reallocates vision and language compute with visual token compression, tiling, positional tokens, long context, and data mixtures. Parameter count and vision input cost are separate; one rearrangement operator is not the full recipe.
Illustrative example: A smaller LM still pays vision encoding and context costs if given many high-resolution visual tokens.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
Where does pixel shuffle put neighborhood information when it reduces visual token count?
SmolVLM: Redefining small and efficient multimodal models
Show answer and example
Answer: It concatenates nearby spatial tokens along channels and projects them through an MLP to the LM. It trades attention length for wider local representations rather than dropping patches. Fine text and spatial relations still require task-specific verification.
Illustrative example: Four adjacent patches merge into one wider vector, quartering token count but coarsening the LM's spatial granularity.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
What does space-to-depth preserve compared with averaging four patches?
SmolVLM: Redefining small and efficient multimodal models
Show answer and example
Answer: Averaging erases independent patch positions; space-to-depth concatenates them in fixed spatial order so later projection can distinguish them. It trades shorter sequences for wider input tokens. Fine recognition still depends on projector capacity and training.
Illustrative example: Swapping black upper-left and white lower-right blocks can preserve their average but change their ordered concatenation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
Does the local r=2 check include SmolVLM's high-resolution and long-video recipe?
SmolVLM: Redefining small and efficient multimodal models
Show answer and example
Answer: No. It uses 32×32 clothing images and 2×2 space-to-depth without high-resolution tiling, video sampling, long context, learned position tokens, or a SmolLM decoder. Results only check the rearrangement and projection.
Illustrative example: Reducing 16 patches to four verifies tensor shapes, not cross-frame localization or 8,000-token context.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
Does UMRM's unsupervised label mean no labels are used anywhere?
Unsupervised Multimodal Reward Modeling
Show answer and example
Answer: No. Caption perturbations construct preference pairs without human pairwise annotation. Layer selection uses a labeled development set, and bootstrapping uses cheap verifier labels. Preference construction, model selection, and deployment-utility supervision differ.
Illustrative example: Change “two cups” to “three cups” automatically, but validation labels can still determine the best layer.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
Why does UMRM compare all B² positive–negative pairs?
Unsupervised Multimodal Reward Modeling
Show answer and example
Answer: All positives and negatives in the batch contribute ranking constraints instead of isolated pairs. Pool only response tokens so prompt/image features do not swamp response distinctions. Synthetic negatives define training signals but do not cover every real policy hallucination.
Illustrative example: Two positives and two negatives give four comparisons rather than two matched pairs, without creating human preference labels.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
Why can caption-perturbation accuracy and real-policy selection utility move in opposite directions?
Unsupervised Multimodal Reward Modeling
Show answer and example
Answer: Synthetic negatives may differ from actual policy errors. A reward head can learn artifacts of artificial editing. Bootstrapping on policy candidates with a verifier changes the supervision and may improve Best-of-N even if a reward benchmark falls.
Illustrative example: Spotting swapped colors does not ensure that a reward head can select a faithful response when policy candidates omit objects.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
Should UMRM change the image to match a perturbed negative description?
Unsupervised Multimodal Reward Modeling
Show answer and example
Answer: No. The external sampler edits the response while holding the image fixed. It can change a span, splice tails, shuffle clauses, or use a cross-image caption with default probability 0.15. Swapping the image changes the contrast. Do not pool image or prompt tokens into the response representation.
Illustrative example: Change “two dogs” to “three dogs,” but keep the original image with two dogs.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Basics
How does interleaved WeMM input differ from a single image vector?
WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report
Show answer and example
Answer: One input can combine text, images, video, and document segments in order before encoding. It represents combinations rather than indexing each modality in isolation. A shared space does not remove the need to train and validate combinations.
Illustrative example: A product photo plus “same design in blue” differs from either the photo or the color term alone.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Core mechanism
Why align at scale before curated relevance and cross-scale distillation in WeMM?
WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report
Show answer and example
Answer: Initial alignment builds a shared space; refined relevance addresses items that look similar but fail the query. Distillation and Matryoshka representations carry semantics across sizes. Closeness alone is not business relevance or correctness.
Illustrative example: Red shoes match “red items,” but “shoes for rain” needs relevance supervision beyond appearance.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Method comparison
How does WeMM cross-scale KD differ from supervising only the largest embedding?
WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report
Show answer and example
Answer: It sums teacher-to-student KL at designated Matryoshka sizes, so smaller outputs receive explicit signals. A common semantic target does not guarantee identical quality or arbitrary truncation. Compare retrieval quality and cost at specified sizes.
Illustrative example: Aligning the full vector does not guarantee the first half works well; the multiscale loss explicitly supervises that distribution.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Cross-modal alignment: representations, connectors, and supervision
Implementation details
Can local MovieLens Matryoshka results replace WeChat A/B?
WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report
Show answer and example
Answer: No. Public content and collaborative views test shared space, refinement, and truncation without the original checkpoint or private multimodal pairs. MMEB and WeChat results belong to the authors' system. Local low-dimensional proxies cannot reproduce business effects or rank on the same table.
Illustrative example: Compare Recall for four- and 16-dimensional proxy vectors locally; do not project it to video-feed percentage gains.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Topic guide and reading order
Start with this question¶
How do images, speech, and text connect to the same task? Does the model learn similarity or learn to generate answers?
Takeaway¶
CLIP uses separate image and text encoders and a paired contrastive objective to learn shared representations for zero-shot classification and retrieval. BLIP-2 freezes the visual encoder and LLM and uses a Q-Former with a small number of queries to extract visual evidence, connecting vision and language generation in stages. Instruction tuning, distillation, reward modeling, and task-specific evaluation change supervision and readout. Track the inputs, objective, and output interface separately.
Example¶
Finding the image that best matches “red umbrella” is a suitable embedding comparison. Reading the second line of small print needs fine visual evidence and a text-output interface. Connecting an image to an LLM does not automatically give it OCR, and compressed visual queries may lose details.
Common pitfall¶
Retrieval similarity is not a probability of factual correctness. Matching words does not prove the image was used. Check negative controls such as shuffled images and distinguish natural image–text pairs, speech accents, and video domains. Small-image runs, random fixtures, and public benchmarks support different claims.
Reading order¶
Start with CLIP's dual encoders, normalization, and contrastive loss. Then read BLIP-2's Q-Former and two-stage connection to distinguish shared embeddings from language generation. Choose later work on instruction tuning, distillation, rewards, or evaluation to match your task.
- Learning Transferable Visual Models From Natural Language Supervision · Original paper
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models · Original paper
Check your understanding¶
Can a high CLIP retrieval score establish that the model reads all text in an image?
Show suggested answer
No. Contrastive pairing mainly constrains overall image–text compatibility. Small text needs enough visual resolution, local representations, and decoding supervision. Evaluate OCR or the relevant fine-grained task separately and verify that results change with image content.