When Synthetic Data Hurts: On Catastrophic Forgetting in Skill Retrieval for LLM Agents

简体中文

L1 real-route anchor, conflict penalty, and replay; no 0.6B training on 34,396 skills.

Paper information

Field	Details
Paper	When Synthetic Data Hurts: On Catastrophic Forgetting in Skill Retrieval for LLM Agents
Institution	Manulife (first author).
First public release	2026-09-09
Author code	Author EMNLP 2026 repository.
Local adapter or method key	skill-retention
Local reproduction code	src/auto_research/agent_research/latest_20260914.py

Paper method and evidence

Combine real-example replay with embedding anchoring/LwF-style regularization so synthetic skill tuning preserves real and OOD routing.

flowchart LR
  S[Synthetic skill examples] --> U[Retrieval adaptation]
  R[Real replay and embedding anchors] --> U
  U --> O[Preserve real and OOD routing]

Original paper figure

The original figure documents synthetic-data adaptation and retention. From the original paper; copyright remains with the authors.

Reported results

The paper reports +13.98% retrieval in the synthetic domain with preserved OOD performance using a 0.6B Qwen retriever/reranker. Not local or production evidence.

Local reproduction

Three-seed metrics use the same public fixtures, budget, and seeds. The deterministic reference exposes decisions and state transitions; tests verify invariants and boundaries. These are not capability or production results.