RoFormer: Enhanced Transformer with Rotary Position Embedding¶

Fidelity: core mechanism. Paper conclusions, public-data results, and omitted work are reported separately.

Paper information¶

Field	Details
Paper	arXiv 2104.09864
Organization	Zhuiyi Technology
First public release	2021-04-20 (arXiv v1)
Authors' code	Public implementation
Adapter / method	`rope`
Local reproduction code	`src/auto_research/reproductions/rope/`

Original paper¶

Motivation and method¶

Rotate two-dimensional Q/K subspaces of each attention head by position, so their dot product depends naturally on relative offsets.

flowchart LR
 A["Public input or history"] --> B["RoPE core path"]
 B --> C["Paper-specific state or objective"]
 C --> D["Matched-budget evaluation"]

Key figure from the paper¶

Figure 1: Overview of the paper's method and key components. From the original paper; copyright remains with the authors.

Core equations¶

\[ q_m=R_mW_qx_m,\ k_n=R_nW_kx_n,\quad q_m^\top k_n=x_m^\top W_q^\top R_{n-m}W_kx_n. \]

Paper results¶

The paper reports improvements over absolute position encodings on several long-text classification and language-modeling tasks.

Local reproduction¶

Local comparison: Same-budget llama_modern baseline and rope treatment; relative perplexity +0.00%.

WikiText-2, 12 steps, width 64, two layers, seed 42: PPL 421.18 → 421.18 (+0.00%). Parameters, tokens, optimizer, and steps are matched.

auto-research reproduce --paper rope --dataset-dir data --seed 42
auto-research evolve --model micro-llm --dataset wikitext-2 --direction "组合 rope 与已安装论文算子" --generations 2 --population 4

See the fixed metrics.

Reproduction boundaries¶

The local path rotates Q/K in the complex plane for each head and incorporates relative phase into their dot product. A width-64 WikiText-2 model stands in for Chinese RoFormer pretraining. Local values do not represent the paper's large model, private data, production traffic, or custom kernels, and must not be conflated with its gains.