Training budgets: change the data, the update direction, or the comparison?¶

Updated 2026-11-04 · 30 topic entries / 8 representative comparisons

Better training within one budget can come from data mixtures, per-example editing, update quality, or lower state costs. DoReMi, Mixing Laws, DataOrchestra, Muon, and ECO act at different points. Muon Paired Budget freezes model selection and asks equal-data and equal-time questions separately. ProLand changes LoRA's product-space update geometry; GAVAGAI adds explicit word-region alignment to captioning. They are not the same intervention merely because both affect training.

Mechanisms · Trade-offs · Cross-paper questions · Updates

How to read this card

The table compares representative methods. It is not a shared-benchmark leaderboard. Teaching examples are not measured results.

Mechanism comparison¶

Axes: Change exposure frequency, content, or parameter updates · Online feedback, pilot fitting, or an offline teacher · Costs of preprocessing, development trials, and optimizer state or work · Equal data, wall-clock time, or frozen candidate-search budget

Paper and source	Mechanism	Benefits and trade-offs
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Use small-proxy group DRO on domain excess loss relative to a reference, then transfer the learned mixture to the target.	Assumes proxy domain utility transfers. A local two-domain unigram checks weighting, not large-target training.
Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Run pilot mixtures, fit domain loss as a function of mixture and scale, and search previously untrained mixtures.	Pilots cost compute and extrapolation can fail. A small local proxy is not a true multiscale curve.
DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Choose Drop, Untouch, or Clean per chunk, then plan denoising, surface repair, or pedagogical enhancement when cleaning.	Edits supervision rather than just frequency; excessive deletion or rewriting can hurt. Preserve the negative local rules-based result instead of treating it as large-model cleaning.
Muon is Scalable for LLM Training · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Approximate a momentum matrix's polar factor for hidden matrix updates; other parameters still use auxiliary AdamW.	Changes update geometry but adds orthogonalization and auxiliary optimizer costs. Short local runs do not reproduce efficiency at scale.
When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 代码映射与边界	Match initialization, data order, and frozen development candidates, and compare data AUC, time AUC, and tuning regret.	Comparison protocol, not a new optimizer. Local synthetic trajectories train neither Muon nor AdamW.
ECO: Quantized Training without Full-Precision Master Weights · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 本地复现与成本判断	Update low-precision weights directly and inject requantization error into existing momentum to avoid persistent master weights.	Adam needs per-element effective steps; format and rounding matter. Buffer-free rules are not exactly equivalent in general. No local implementation/training; full experiments were too costly.
ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-Tuning · Original source Sections in the Chinese source: 核心公式, 检查与负控, 成本判断与复现边界	Reproject momentum and compute a spectral direction in the product space BA; coordinate the factors with horizontal DB and independently restore B's orthogonality through Landing.	Not independent Muon on A/B. A smaller core saves state but does not remove ill-conditioned pseudoinverses. Local coverage verifies exact thin-factor updates only; full multi-model fine-tuning is too costly and not reproduced.
GAVAGAI: Referential Alignment for Infant-Scale Vision-Language Models via Balanced Optimal Transport · Original source Sections in the Chinese source: 核心公式, 检查与负控, 成本判断与复现边界	Use Nothing for nonreferential words and visual-column upper capacities for regional competition, then add fixed-plan alignment cost to caption CE.	Changes supervision rather than the optimizer. Eight scaling steps do not guarantee row-marginal convergence, and v1 leaves any extra Nothing-hinge gradient unspecified. Local coverage claims neither full anti-collapse nor SAYCam training.

Trade-off analysis¶

Sampling a domain more often differs from rewriting its examples¶

DoReMi adjusts domain frequency using excess loss, and Mixing Laws predicts mixtures from pilots. DataOrchestra chooses whether and how to edit each chunk, changing supervision. More cleaned text or lower mean loss does not guarantee target gains. Isolate data volume, content, and preprocessing cost; specify whether raw tokens, processed tokens, or steps are fixed.

Sources: DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining · Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance · DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

The budget can reverse the conclusion¶

Muon changes update direction; ECO changes persistent state and quantized updates. Both can change step time. Paired Budget separates equal-data and equal-time comparisons with frozen development choices, avoiding a new champion per report. Count auxiliary AdamW, preprocessing, and pilots. Formulas and synthetic curves verify accounting only, not actual training efficiency.

Sources: Muon is Scalable for LLM Training · ECO: Quantized Training without Full-Precision Master Weights · When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark

Separate how parameters update from what the model must learn¶

ProLand keeps the supplied loss while changing LoRA updates through product-space spectral directions and Landing. GAVAGAI adds referential variables to the objective and addresses supervision mismatch with Nothing and capacities. Descent and feasibility recovery do not validate a new alignment loss; feasible small-matrix transport does not establish real-image grounding gains. A future combination should fix data, backbone and budget, then ablate objective changes independently of update geometry. Both are L1 components here; no costly joint training is launched.

Sources: ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-Tuning · GAVAGAI: Referential Alignment for Infant-Scale Vision-Language Models via Balanced Optimal Transport

Cross-paper questions¶

Try answering before expanding the explanation. Each question relates at least two papers.

Why can DoReMi upweighting and DataOrchestra cleaning change a difficult domain in opposite ways?¶

Show answer and example

Answer: DoReMi samples more from domains underfit relative to a reference, assuming their contents are useful. DataOrchestra may drop, retain, or rewrite chunks. Upweighting can repeat noise, and aggressive cleaning can remove clean signal. Diagnose the issue and compare equal budgets.

Illustrative example: Rare correct equations can merit more exposure despite high excess loss, whereas corrupted text should be cleaned rather than sampled more.

Sources: DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining · DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

How do DoReMi and Data Mixing Laws differ in assumptions when choosing large-model mixtures from small runs?¶

Show answer and example

Answer: DoReMi assumes proxy excess-loss dynamics identify worthwhile domains. Mixing Laws assumes pilot proportions and scales sufficiently fit a loss surface for untested combinations. Proxy-target differences can break either. Record pilot costs and evaluate the target independently rather than reporting fitted scores as training.

Illustrative example: A proxy may underfit code that a large model already understands, and fitted curves may fail at extreme mixtures.

Sources: DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining · Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance

Why doesn't lower Muon loss at equal tokens prove a win under fixed training time?¶

Show answer and example

Answer: Tokens control data exposure, not orthogonalization, auxiliary AdamW, or throughput. Compare validation on the shared observed time range using frozen development choices. Measure ECO's state savings separately, since smaller state need not mean faster steps.

Illustrative example: A more effective but twice as slow update may lose to one that sees more data in the same hour.

Sources: Muon is Scalable for LLM Training · When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark · ECO: Quantized Training without Full-Precision Master Weights

Why can't one decreasing loss classify ProLand and GAVAGAI as the same training improvement?¶

Show answer and example

Answer: ProLand changes parameter updates and feasibility restoration for a given objective; GAVAGAI changes the objective with word-region assignments and Nothing. Their losses may not even be comparable. Fix task and data, then separately test update equivalence, transport constraints, independent validation performance and total costs. Small-matrix descent is not capability evidence.

Illustrative example: Teaching example: Adding a potentially negative alignment cost lowers CE-plus-cost numerically without necessarily improving captions. Likewise, reducing MSE with ProLand does not show that words were grounded in images.

Sources: ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-Tuning · GAVAGAI: Referential Alignment for Infant-Scale Vision-Language Models via Balanced Optimal Transport

Update log¶

Updates record new approaches, changed trade-offs, and follow-up evidence.

2026-11-04 · Add ProLand and GAVAGAI, separating product-space update geometry from new alignment supervision. Document independent references, finite-iteration and gradient boundaries; both full training experiments are too costly and not reproduced.

Affected: ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-Tuning · GAVAGAI: Referential Alignment for Infant-Scale Vision-Language Models via Balanced Optimal Transport

2026-10-27 · Initial version: compared learned domain mixtures, pilot curve extrapolation, example-level editing, matrix optimization, and low-precision states under paired-budget comparisons.

Covers 28 topic entries; see the full list below.

All 30 topic entries

The full catalog follows. The main comparison covers the representative papers listed above.

Paper-by-paper questions · All deep dives