简体中文 · English
NoteLLM: Adding collaborative signals to a retrievable LLM¶
Fidelity: full core pipeline. Runs a note-compression special token, behavioral co-occurrence GCL, category-generation CSFT, and I2I retrieval from the compressed state. T5-small and MovieLens replace the 7B LLM and Xiaohongshu notes.
Paper information¶
| Field | Details |
|---|---|
| Paper | arXiv 2403.01744 |
| Institution | Xiaohongshu |
| First public release | 2024-03-04 (arXiv v1) |
| Author code | No official or author-released code found in the paper (checked 2026-08-09). |
| Local adapter or method key | notellm |
| Local reproduction code | src/auto_research/reproductions/notellm/ |
Original paper summary¶
Background and main changes¶
Text-only embeddings lack collaborative signals, and contrastive learning alone may harm LLM generation. NoteLLM uses one compression prompt to distill content into a special token. GCL pulls co-read notes closer together; CSFT generates hashtags or categories from the same compressed state.
flowchart LR
N["Note title/content/category"] --> P["Compression prompt + special token"]
P --> L["Autoregressive LM"]
L --> Z["Compressed state"]
Z --> G["GCL: co-click contrast"]
Z --> S["CSFT: category generation"]
Z --> R["I2I vector retrieval"]Key figure from the paper¶
Figure 2: The core architecture, its main modules, and their connections. From the paper; copyright belongs to the authors. Click for the source.
Core equations¶
Offline and online results¶
Paper Recall@100: SentenceBERT 70.72%, RepLLaMA 83.63%, NoteLLM 84.02%. In a one-week Xiaohongshu I2I A/B test, CTR rose +16.20%, comments +1.10%, weekly publishers +0.41%, and comments on new notes +3.58%.
Local reproduction¶
Local comparison: The baseline uses frozen compression-state embeddings; the experimental arm uses NoteLLM GCL+CSFT embeddings. NDCG@10 goes from 0.01142 to 0.01224 (+7.15%). This ablates representation training, not an end-to-end ranking gain against DIN.
| Model | Hit@10 | NDCG@10 |
|---|---|---|
| Frozen compression state | 0.01931 | 0.01142 |
| NoteLLM GCL + CSFT | 0.02110 ± 0.00223 | 0.01224 ± 0.00037 |
Hit@10 rises +9.26%, NDCG@10 +7.15%, with NDCG up on 3/3 seeds. Public data provides only titles and genres; category generation is easier than generating real hashtags. The result supports only the local trend for the joint objective. See metrics/movielens-100k-seeds42-44.json.