Hymba: A Hybrid-head Architecture for Small Language Models¶

Core hybrid-head mechanism; local and original results are separate.

Paper information¶

Field	Details
Paper	arXiv 2411.13676
Institution	NVIDIA.
First public release	2024-11-20 (arXiv v1)
Author code	NVlabs/hymba.
Local adapter or method key	`hymba`
Local reproduction code	`src/auto_research/reproductions/hymba/`

Paper method and evidence¶

Run attention and a state-space branch in parallel within one layer. An input-dependent gate combines exact local retrieval with linear long-range state.

flowchart LR
  X[Layer input] --> A[Causal attention]
  X --> S[State-space branch]
  A --> G[Input-dependent gate]
  S --> G

Original paper figure¶

The original figure shows hybrid attention and state branches. From the original paper; copyright remains with the authors.

Core equation¶

\[ H'=H+g(X)\odot A(X)+(1-g(X))\odot SSM(X). \]

Reported results¶

Hymba-1.5B reports a better accuracy/cache/throughput tradeoff among similar-size models.

Local reproduction¶

On WikiText-2 at 12 steps, 64 dimensions, two layers and seed 42, matched llama_modern → hymba PPL is 421.18 → 457.10 (+8.53%). Parameter, token, optimizer, and step budgets match.

auto-research reproduce --paper hymba --dataset-dir data --seed 42
auto-research evolve --model micro-llm --dataset wikitext-2 --direction "Combine hymba with installed paper operators" --generations 2 --population 4

Fixed metrics. Actual causal attention and depthwise-convolution state branches are gated per layer. No 1.5B run, meta tokens, fused kernel, or paper-scale efficiency result.