Fast Inference from Transformers via Speculative Decoding¶

Core mechanism on public data, not paper-scale kernel latency.

Paper information¶

Field	Details
Paper	ICML 2023 Oral
Institution	Google Research.
First public release	2022-11-30 (arXiv v1)
Author code	Google Research speculative_decoding.
Local adapter or method key	`speculative-decoding`
Local reproduction code	`src/auto_research/reproductions/speculative_decoding/`

Paper method and evidence¶

A smaller draft model proposes multiple tokens; the target verifies a whole block. On rejection, sample from the corrected residual distribution, preserving the target distribution.

flowchart LR
  D[Draft proposes a block] --> T[Target verifies block]
  T --> A[Accept eligible prefix]
  T --> R[Sample residual on rejection]

Original paper figure¶

The original figure shows drafting and exact target verification. From the original paper; copyright remains with the authors.

Core equation¶

\[ a(x)=\min(1,p(x)/q(x)),\quad p'(x)\propto[p(x)-q(x)]_+. \]

Reported results¶

The paper reports 2–3× speedup on T5-XXL with the same output distribution.

Local reproduction¶

Target-only greedy versus four-token draft and exact verification: local target_calls 160.0000 → 40.0000 (-75.00%). Fixed metrics.

auto-research reproduce --paper speculative-decoding --dataset-dir data --seed 42

Fit small Markov target/draft token models on WikiText-2 and run proposal, verification and rejection fallback. Calls are not T5-XXL GPU latency.