BLIP-2: Connect frozen vision and language models with Q-Former¶
Core query/cross-attention mechanism, not two-stage large-scale pretraining.
Paper information¶
| Field | Details |
|---|---|
| Paper | Paper or original source |
| Institution | Salesforce Research. |
| First public release | 2023-01-30 (arXiv v1) |
| Author code | salesforce/LAVIS. |
| Local adapter or method key | blip2 |
| Local reproduction code | src/auto_research/reproductions/blip2/ |
Paper method and evidence¶
Freeze the image encoder and LLM; train a lightweight Q-Former with learnable queries attending to vision tokens. First learn image–text representations, then project queries as a soft visual prompt for the frozen LLM.
flowchart LR
A[Frozen visual tokens] --> B[Learnable queries cross-attend]
B[Learnable queries cross-attend] --> C[Project to frozen LLM]Original paper figure¶
The original figure shows both stages. From the original paper; copyright remains with the authors.
Core equation¶
Reported results¶
The paper reports +8.7 zero-shot VQAv2 points versus Flamingo-80B with 54× fewer trainable parameters, plus caption, VQA and retrieval results. No industrial A/B.
Local reproduction¶
Matched mean-pooling connector vs four-query Q-Former: test accuracy 46.8%→41.5% (-5.3 points) and visual tokens 16→4 (-75%). Metrics. On 2,000 real Fashion-MNIST images and 120 steps, fewer tokens came with worse accuracy. No ViT-g, OPT/FlanT5, or 129M-image ITC/ITM/ITG training.