Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension¶
Fixed-context NumPy mechanism, not paper-scale training.
Paper information¶
| Field | Details |
|---|---|
| Paper | Paper or original source |
| Institution | Ai2; first author Amanda Bertsch. |
| First public release | 2026-08-10 (arXiv v1) |
| Author code | allenai/olmpool. |
| Local adapter or method key | olmpool-long-context |
| Local reproduction code | src/auto_research/reproductions/olmpool_long_context/ |
Paper method and evidence¶
Use a controlled 7B model pool to isolate compound effects of normalization, GQA, pretraining length, and sliding-window attention on long-context extension.
flowchart LR
A[Control architecture choices] --> B[Extend context]
B[Extend context] --> C[Measure compound effects]Original paper figure¶
The original figure shows architectural sensitivity. From the original paper; copyright remains with the authors.
Core equation¶
\[
\Delta_{LC}=f(N,G,L,S)-f(N_0,G_0,L_0,S_0)
\]
Reported results¶
Three or more unfavorable choices can reduce downstream long-context performance to 47%; 26 OlmPool models are released.
Local reproduction¶
Same seed and 64 examples: recent-window 0.0781, OlmPool mechanism 1.0000, +92.19 percentage points. Metrics, batch index. No full checkpoint training, CUDA kernel, or benchmark.