Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance¶
Core mixture prediction, not original RedPajama scale.
Paper information¶
| Field | Details |
|---|---|
| Paper | Paper or original source |
| Institution | University of Cambridge / Shanghai AI Laboratory. |
| First public release | 2024-03-25 (arXiv v1) |
| Author code | yegcjs/mixinglaws. |
| Local adapter or method key | data-mixing-laws |
| Local reproduction code | src/auto_research/reproductions/data_mixing_laws/ |
Paper method and evidence¶
Train small pilot mixtures, fit per-evaluation-domain mixing laws and search previously unseen mixtures.
flowchart LR
A[Train pilot domain mixtures] --> B[Fit per-domain laws]
B[Fit per-domain laws] --> C[Select unseen mixture]Original paper figure¶
Figure shows pilot mixtures, fitting and search. From the original paper; copyright remains with the authors.
Core equation¶
\[
L_i(\mathbf p,N)=c_i+\sum_j a_{ij}p_j^{-\beta_{ij}}N^{-\gamma_i},\quad\mathbf p^*=\arg\min_{\Delta} \sum_iw_iL_i.
\]
Reported results¶
The paper uses small-model curves to predict higher-budget RedPajama mixtures, outperforming manual and uniform mixes.
Local reproduction¶
WikiText-2 plus public narrative, matched token/validation protocol, seed 42. Uniform vs learned validation loss 5.6729→5.4758 (-3.48%). Fit per-domain exponential laws to ten pilots and search unseen grid points. Fixed metrics. Unigram proxy is not a multi-model RedPajama scaling curve.