CRITIC¶

Use tool feedback to verify and revise a first draft.

Paper information¶

Field	Value
Paper	CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Affiliations	Microsoft / Tsinghua University
First public release	2023-05-19
Authors' code	Author code.
Adapter / method	`critic`
Local code	`src/auto_research/agent_research/`

Original paper¶

Background and contribution¶

Pure reflection can repeat a model's original mistake. CRITIC adds observable evidence from external tools, such as search and code execution, to each critique-and-revision iteration.

flowchart LR
  D[First draft] --> T[External tool check] --> F[Errors and test feedback] --> C[Tool-based critique] --> R[Revision] --> T

Key figure from the paper¶

Original Figure 1: CRITIC. From the paper; copyright belongs to the authors. Not a local result.

Core mechanism¶

\[ y^{(t+1)}=\operatorname{Revise}\!\left( x,y^{(t)},\operatorname{Tool}(y^{(t)})\right). \]

Paper results¶

The paper reports consistent gains in open QA, math program synthesis, and toxicity reduction. No production A/B.

Local reproduction¶

Write a defective patch, run actual unittest, then revise and retest from the traceback. Record edit, command, status, and output. See stable metrics.

auto-research agent-eval --method critic --benchmark swebench-local \
  --episodes 12 --seed 42

Reproduction limits¶

The environment actually executes feedback and edits, but is a controlled code test. No search engine, toxicity classifier, or general LLM critic.