Training budgets: data mixtures, objectives, and optimizers¶
简体中文 · English
Foundation models / 120 questions · 30 papers
Try answering each question before opening the explanation. The bank grows as papers and questions are added.
Read this topic's comparisons, analysis, and cross-paper Q&A
120 questions · 30 papers
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why does GAVAGAI need both Nothing and region capacities?
Show answer and example
Answer: Nothing lets nonreferential words avoid forced assignment to backgrounds; capacities stop a few persistent regions from absorbing many words. Capacity without Nothing still forces mass onto visual regions. Nothing without capacities cannot limit concentration on one background. They address different failures.
Illustrative example: Teaching example: An utterance contains hungry and ball while the image contains a ball and hand. Hungry can go to Nothing and ball to the ball; a hand capacity also prevents many unrelated words from accumulating there. These assignments are illustrative, not measured.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Despite Balanced OT in the title, why not fix every visual column marginal by equality?
Show answer and example
Answer: Eq. 1 sets each word-row sum to 1 but only bounds visual column sums by b; Nothing is uncapped. Thus the dual update is v=min(1,b/(Kᵀu)). Requiring every column sum to equal b would assign mass to unnamed regions, changing the feasible set and interpretation.
Illustrative example: Teaching example: If neither of two words names a visible object and a visual column has capacity .8, nearly all mass 2 may legally go to Nothing and almost none to the region. Do not force reference just to spend the .8 capacity.
Sections in the Chinese source: 核心公式, 检查与负控. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
How does GAVAGAI reduce to row softmax when capacities are inactive, and does that erase the method's distinction?
Show answer and example
Answer: If every visual capacity is at least the word count n, column multipliers remain 1, giving row softmax including Nothing. Removing Nothing then recovers forced visual reference. This degeneration is a useful check, not equivalence under active capacities or evidence of SAYCam improvement.
Illustrative example: Teaching example: With 3 words and capacity 3 per column, no visual column can exceed its cap. Changing that cap to .4 couples row and column scalings as words compete for the region.
Sections in the Chinese source: 核心公式, 检查与负控, 论文离线与线上效果. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Why not renormalize rows after eight Sinkhorn steps, and why detach the plan in alignment loss?
Show answer and example
Answer: Final column scaling enforces upper capacities, but finite-iteration row sums can differ from 1. Row renormalization can exceed capacities again, so expose residuals. Eq. 3 fixes P and backpropagates only through C, giving gradient λP. V1 does not specify another differentiable route for the Nothing hinge; local coverage reports a fixed-plan value, not full anti-collapse training.
Illustrative example: Teaching example: Renormalizing a row summing to .8 multiplies its visual mass by 1.25 and can violate an already saturated column. With λ=.1, cost gradient is .1P, not differentiation through eight scaling iterations.
Sections in the Chinese source: 核心公式, 检查与负控, 成本判断与复现边界. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why does ProLand optimize while B is temporarily nonorthogonal instead of projecting it back every step?
ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-Tuning
Show answer and example
Answer: It separates the objective direction for BA from Landing restoration of B's orthogonality. BᵀDB=0 makes the objective update perturb the Gram error only at second order, while an independent correction restores feasibility. Per-step retraction changes the trajectory and is not the same algorithm; temporary infeasibility does not imply stability at arbitrary step sizes.
Illustrative example: Teaching example: A one-dimensional basis with BᵀB=1.08 can move horizontally and then subtract ρB×0.08. Direct normalization to 1 follows a different path. This is not a GLUE result.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why reproject momentum into the current product space rather than retain the old dense direction unchanged?
ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-Tuning
Show answer and example
Answer: Changing B and A changes the space realizable by their first-order variations. ProLand projects the previous EMA through the current Π before mixing the current projected gradient, keeping momentum realizable. The thin-factor algorithm reconstructs this quantity from gradients of the same loss; it is not ordinary independent Muon momentum for A and B.
Illustrative example: Teaching example: If old B spans the first coordinate and new B turns toward the second, an old-space direction may no longer be realizable unchanged. Compute Π using the new PB and PA before updating the EMA.
Sections in the Chinese source: 核心公式, 检查与负控. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
How does polar on A and B separately differ from a product-space polar direction realized through factor updates?
ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-Tuning
Show answer and example
Answer: Polar is nonlinear and the factors have a scaling gauge. Independent spectral updates do not ensure their combined effect matches the selected product-space direction. ProLand constructs DA=B†Y and DB=(I−PB)YA† from one Y, ensuring BDA+DBA=ΠY and BᵀDB=0.
Illustrative example: Teaching example: Multiplying B by 2 and dividing A by 2 preserves BA but changes independent factor-gradient scales and geometry. Calling polar in both paths does not make their ΔW updates equivalent.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Why not update A first and use new A for B, or multiply ρ by the learning rate again?
ProLand: A New Infeasible Manifold Optimization Method for LoRA Fine-Tuning
Show answer and example
Answer: Algorithms 1/2 compute DA, DB and Gram error from one old factor pair and apply them simultaneously. Sequential updates add coupling terms. Landing's ρ is an independent step size, with no additional γ. Compact polar must preserve zero singular directions, and numerical ill-conditioning near rank changes must remain explicit.
Illustrative example: Teaching example: With γ=.01 and ρ=.05, the Landing coefficient is .05, not .0005; the latter weakens restoration 100-fold. A full UVᵀ on a zero core can also invent updates under zero gradients.
Sections in the Chinese source: 核心公式, 实现与运行, 检查与负控. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why does state quantization affect later steps?
Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization
Show answer and example
Answer: First and second moments recur, so stored errors enter optimization after decoding at the next step. Control their effects on preconditioning and future updates, not merely bytes.
Illustrative example: Repeated downward rounding of v can inflate effective learning rates across steps.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why control second-moment rounding in preconditioner space in 4-bit AdamW?
Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization
Show answer and example
Answer: Bias correction, square root, and reciprocal nonlinearly transform moment error into update error. ZIP-SR uses preconditioner values for stochastic rounding; ZE-EDEN handles code points and scales. Error persists through state recurrence. Use working moments for the update before packing.
Illustrative example: Small moment differences can greatly change reciprocal-root steps, so raw moment MSE misses the effective gradient scale.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
How do ZIP-SR and ZE-EDEN store second moments differently?
Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization
Show answer and example
Answer: ZIP-SR stochastically rounds in preconditioner space. ZE-EDEN excludes zero code points and calibrates block scales. Compare training loss and actual persistent bytes, not a generic four-bit rounding table.
Illustrative example: Randomly choosing adjacent codes differs from rescaling blocks to avoid zero codes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Should current working moments be quantized before updating parameters?
Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization
Show answer and example
Answer: No. Decode old state, form new working moments, update using their unquantized values, then pack for persistence. Reversing the order injects error into the current step.
Illustrative example: Use v to precondition this step before compressing it for the next.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why use different updates for different parameter types?
AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models
Show answer and example
Answer: Hidden matrices, tied vocabulary, and one-dimensional parameters have different geometry and support. Use Muon, support-aware capped vocabulary directions, and RMS respectively.
Illustrative example: A sparsely visited input embedding row differs from a dense hidden matrix.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why not apply ordinary Muon to tied embeddings?
AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models
Show answer and example
Answer: Tied vocab weights support sparse input lookup and dense output classification with different gradient geometry. AF-Muon uses spectral updates for hidden matrices, support-aware capped directions for vocab, and RMS for vectors. Output gradients can touch rows absent from the input.
Illustrative example: An unseen input word can still receive classification gradient as an output candidate.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
How does support-aware capped direction differ from unit-normalizing every vocab row?
AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models
Show answer and example
Answer: It preserves zero support and caps row magnitude rather than forcing every nonzero row to equal length. Unit normalization can amplify small gradients and mishandle unsupported rows.
Illustrative example: Do not enlarge a small gradient merely to give it the same norm as a large row.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
How avoid NaN on zero-gradient vocab rows?
AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models
Show answer and example
Answer: Branch explicitly: zero gradients get zero directions. Multiplying by zero after computing g/‖g‖ cannot clear NaN. Local checks verify support and cap, not paper-scale pretraining.
Illustrative example: A row absent from both input and output gradients must not compute 0/0.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why does BitSparse suit exact ReLU zeros?
BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks
Show answer and example
Answer: A bitmask stores positions, positive values are packed, and tile prefixes locate data. Reconstruction is lossless without approximating nonzero values.
Illustrative example: Two positives among ten positions need two values and locations, not ten floats.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
How can BitSparse save activations losslessly without recomputing forward?
BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks
Show answer and example
Answer: ReLU-family activations contain exact zeros. Store only nonzero values, a bit per position, and tile prefixes. Reconstruct for backward without quantizing values or rerunning forward. Savings depend on zeros and metadata.
Illustrative example: For eight entries with six zeros, store eight bits and two original-precision values.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
How does lossless packing differ from checkpointing or quantization?
BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks
Show answer and example
Answer: Checkpointing trades recomputation for memory; quantization trades precision. BitSparse exploits exact zeros. Sparse metadata and total training memory still matter.
Illustrative example: With few zeros, metadata can offset packing gains.
Sections in the Chinese source: 背景与主要改动, 结果与独立对照. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Why not use one byte of bool per mask element?
BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks
Show answer and example
Answer: The format uses one bit per position plus tile prefix and positives. Byte masks change storage accounting. Check padding, zero sign, and backward formulas such as the 2H factor in ReLU².
Illustrative example: A 32-position mask should occupy around four bytes, not 32.
Sections in the Chinese source: 核心公式, 数据协议与运行. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
What prunable structure is BMuon designed to learn?
BMuon: Budgeting Muon's Spectral Geometry for Structured Compressibility
Show answer and example
Answer: It couples MLP hidden channel input rows and output columns under a common budget. This is optimizer geometry, not extra gates or arbitrary magnitude pruning.
Illustrative example: Consider both edges of a channel together.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why share BMuon budget across MLPs and apply it to input rows and output columns?
BMuon: Budgeting Muon's Spectral Geometry for Structured Compressibility
Show answer and example
Answer: One hidden channel is an input row and output column pair. A shared budget reallocates compressibility across layers and guides both sides' Muon geometry. It shapes training updates, not forward gates or post-hoc independent pruning.
Illustrative example: A 100-channel budget need not split 50/50, but paired edges use the same allocation.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
How does global budgeting differ from equal per-layer retention?
BMuon: Budgeting Muon's Spectral Geometry for Structured Compressibility
Show answer and example
Answer: All blocks compete for total K; fixed ratios preassign capacity per layer. Equal total channels can have different allocations.
Illustrative example: One layer can keep more while another keeps less instead of pruning 50% everywhere.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Can BMuon reuse the old polar update after an MM budget update?
BMuon: Budgeting Muon's Spectral Geometry for Structured Compressibility
Show answer and example
Answer: No. Use old allocation to compute MM scores, then recompute bilateral polar directions using the new shared allocation and one multiplier. Missing either paired gradient must fail rather than silently decouple.
Illustrative example: A channel moving 0.2→0.8 needs directions based on new μ.
Sections in the Chinese source: 核心公式, 实现与运行. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why train small pilots across mixtures?
Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
Show answer and example
Answer: Pilots observe each domain and support fitting mixture and scale effects. This reduces exhaustive runs but depends on coverage and modeling assumptions.
Illustrative example: Ten pilot mixes can shortlist others, not turn untrained points into measurements.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
How do Data Mixing Laws search untrained mixtures, and what limits them?
Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
Show answer and example
Answer: Small matched runs observe domain losses, a fitted relation predicts mixture/scale performance, and a weighted objective searches it. This avoids exhaustive large runs but depends on pilot coverage and assumptions. Keep final test isolated.
Illustrative example: Use pilots at 20%, 50%, and 80% code to select a few intermediate candidates; predictions are not measured large-model results.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
How does fitted mixture search differ from measuring every mix?
Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
Show answer and example
Answer: Fitted predictions save compute but can have approximation and extrapolation error. Direct runs give measured outcomes at higher cost. Independently validate a predicted optimum.
Illustrative example: A predicted best 70/30 mix has not necessarily trained at target scale.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
How keep test outcomes out of mixture selection?
Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
Show answer and example
Answer: Fit and choose domain weights/candidates under prespecified train/validation rules. Test once at the end. Mixtures are nonnegative and sum to one. A unigram proxy is not RedPajama scaling.
Illustrative example: Do not retune w_i after seeing test domain losses.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why make chunk-level data decisions?
DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data
Show answer and example
Answer: Chunks differ in noise and quality. The orchestrator chooses Drop, Untouch, or Clean and organizes operations or instructions only when needed.
Illustrative example: Preserve a clean explanation while removing or repairing corrupted text.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why allow Drop and Untouch instead of applying all three stages to every chunk?
DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data
Show answer and example
Answer: Clean text can be damaged by costly rewrites. Choose Drop, Untouch, or Clean; only Clean plans noise pruning, surface correction, pedagogical augmentation, and per-chunk instructions. Plans are executed and verified. Optimize quality versus compute, not cleanup count.
Illustrative example: Leave a clean textbook alone, prune navigation on a web page, and drop a fragment missing content.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
Why preserve Untouch in sample-level orchestration?
DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data
Show answer and example
Answer: It explicitly allows no processing, avoiding cleaning rate as a false quality goal. Per-sample plans select stages; one corpus pipeline can overrewrite and waste tools.
Illustrative example: Rewriting correct code may consume compute and damage content.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Can raw teacher plans be used as final SFT targets without verification?
DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data
Show answer and example
Answer: No. Execute tools and verify results; correct failed rewrites up to a limit and remove stages with lost content or insufficient change. Only verified evolved plans train the orchestrator.
Illustrative example: A format repair that deletes content must not remain a success example.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why use linear regression to analyze pre- and post-training?
Show answer and example
Answer: Controlled linear attention exposes how covariance, initialization, and supervision affect weight prediction and permits closed-form references. It is not empirical evidence for full LLM training.
Illustrative example: Change task covariance in a small matrix and isolate how initialization affects fitting.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why isn't OS in the data-quality theory full RL post-training?
Show answer and example
Answer: OS is outcome-supervised regression on final predictions. The comparison is SFT on teacher intermediate states. Linear attention and covariance assumptions enable closed-form analysis, not policy sampling, advantage estimation, or policy gradients.
Illustrative example: Squared error on final weights differs from intermediate-state matching without becoming reward-driven policy optimization.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
How do SFT and OS differ in information supplied?
Show answer and example
Answer: SFT supervises intermediate teacher weights. OS runs an autonomous rollout and supervises only its final prediction, without implying policy gradients.
Illustrative example: Stepwise standard results and a single final weight check are distinct signals.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Why must autonomous rollout not receive true w*?
Show answer and example
Answer: Ground-truth weights belong only in training loss, not prediction inputs. Use public context for rollout, feed targets only to the explicit loss, and preserve initialization's nullspace in closed-form SFT.
Illustrative example: A predictor cannot receive the exact weights it must recover.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why learn target-model mixtures with a small proxy?
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
Show answer and example
Answer: Repeated large runs are costly. Small-proxy group DRO adjusts weights from excess loss, then target training uses that mixture. Transfer needs verification.
Illustrative example: Estimate reference-relative gaps cheaply before allocating large-run data.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why emphasize excess loss rather than maximum raw loss in DoReMi?
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
Show answer and example
Answer: Raw loss mixes irreducible difficulty and underfitting. Subtract reference loss and use group DRO on a small proxy to derive target mixture. This is relative to the reference and data, not a preference for all high-entropy domains.
Illustrative example: A has loss 5 vs reference 4.9; B has 3 vs 2. B's larger excess can get more weight.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
How do dynamic DoReMi weights differ from uniform sampling?
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
Show answer and example
Answer: Uniform probabilities stay fixed. DoReMi exponentiates reference-relative excess-loss updates and renormalizes, responding to underfitting rather than raw loss alone.
Illustrative example: A high loss from an intrinsically hard domain may not imply a large excess.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Why normalize α across domains after each update?
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
Show answer and example
Answer: Exponential updates yield relative weights, not probabilities; normalization sums to one. Keep the same reference. Local two-domain unigram work is not a Pile 280M-to-8B transfer.
Illustrative example: Relative weights 2 and 3 become 0.4 and 0.6.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why remove both master weights and a dedicated error buffer?
ECO: Quantized Training without Full-Precision Master Weights
Show answer and example
Answer: Both consume persistent memory. ECO injects error into existing momentum to accumulate small updates without storing them separately.
Illustrative example: An update too small for one quantization step can still affect later optimization.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
How does ECO preserve small updates without high-precision masters?
ECO: Quantized Training without Full-Precision Master Weights
Show answer and example
Answer: Update quantized weights and feed requantization error into existing momentum without a separate persistent buffer. Compensation depends on step size, elementwise for Adam. The buffer-free derivation is approximate, not exact high-precision equivalence.
Illustrative example: A sub-grid update can leave weights unchanged but influence momentum and later steps.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
Is buffer-free compensation exactly equivalent to storing full adjacent errors?
ECO: Quantized Training without Full-Precision Master Weights
Show answer and example
Answer: No. It uses an approximation that neighboring errors are close, with separate convergence analysis. It is not exact under arbitrary quantization.
Illustrative example: Large differences between adjacent errors can invalidate the approximation.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Why can't ECO use the same scalar coefficient when moving from SGDM to Adam?
ECO: Quantized Training without Full-Precision Master Weights
Show answer and example
Answer: Bias correction and second moments create elementwise effective η_t in Adam; use that for error injection. Record quantization format and scale granularity.
Illustrative example: Two coordinates with different preconditioning cannot share a scalar compensation denominator.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why can activation outliers hurt typical values during quantization?
GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries
Show answer and example
Answer: Fixed bit width must cover the range; large channels enlarge spacing for ordinary values. GaugeQuant learns an orthogonal basis with an outlier objective. Floating-point equivalence need not preserve quantization error.
Illustrative example: A few levels spanning 1, 2, and 1000 struggle to distinguish 1 from 2.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
How can GaugeQuant change coordinates without changing the floating-point function?
GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries
Show answer and example
Answer: Valid symmetries apply an orthogonal basis on one side and its inverse on the other. Quantization depends on coordinate distributions, so equivalent bases can yield different outliers/errors. Learn quantization-friendly bases only at valid locations.
Illustrative example: Spread an extreme channel across several axes while inverse transformation preserves the floating-point output.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
How does GaugeQuant's optimization timing differ from offline calibration?
GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries
Show answer and example
Answer: Train bases and outlier objectives with the LM. Offline methods commonly freeze the model and calibrate from samples later. Bit width alone hides training and data differences.
Illustrative example: Both can be W4A4 while one learns a basis during training and the other calibrates afterward.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
How distinguish functional equivalence, fake quantization, and actual int4 execution?
GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries
Show answer and example
Answer: Check orthogonality and equivalent transforms, then quantization and gradients. STE fake quant can use floating kernels. The local work has no int4 kernel, so no throughput or bandwidth conclusion.
Illustrative example: Sixteen levels do not imply bit packing or int4 instructions.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why not continue training on all internal code unchanged?
Customizing an LLM for Enterprise Software Engineering
Show answer and example
Answer: Extract high-value history and interactions, deduplicate, filter, and control contamination before midtraining and internal-tool post-training. Data curation and evaluation are integral.
Illustrative example: Generated duplicate files and valuable fixes should not receive equal treatment by default.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why include general replay in enterprise code adaptation?
Customizing an LLM for Enterprise Software Engineering
Show answer and example
Answer: Internal examples teach private code, tools, and workflows but can displace general ability. Replay adds a retention constraint; deduplication, filtering, and contamination controls protect the comparison. Validate both sides.
Illustrative example: Learning internal build commands need not preserve general reasoning if training drops all general examples.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
What extra constraints distinguish enterprise midtraining?
Customizing an LLM for Enterprise Software Engineering
Show answer and example
Answer: It includes internal code, interaction, and tools while controlling duplicates and evaluation leakage and preserving general ability. General training loss does not replace internal workflow evaluation.
Illustrative example: Memorizing old fixes can lower loss without helping new internal problems.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
What do the two extreme replay weights verify?
Customizing an LLM for Enterprise Software Engineering
Show answer and example
Answer: λ=0 leaves enterprise loss and λ=1 leaves general replay. This checks branch wiring and coefficient direction. Local tests use no Google private data, Gemini weights, or A/B.
Illustrative example: Enterprise loss dominating at λ=1 suggests reversed weighting or routing.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why is Meta/AMD hardware diversity a co-design report?
Show answer and example
Answer: It changes model, parallelism, memory strategy, GEMM and kernel porting together for recommender training. Performance depends on the combined system, not HIP conversion alone.
Illustrative example: A compiling operator can still use the wrong tile or memory layout.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why doesn't CUDA-to-HIP compatibility establish performance portability?
Show answer and example
Answer: Running code does not align compute units, memory hierarchy, tiles, and parallelism. The work co-designs model, memory, parallel strategy, and kernels and estimates tail stalls. Hardware constants and optima vary.
Illustrative example: A tile filling one GPU's compute units can be limited by shared memory or tail imbalance on another.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
Do predicted tile cycles and end-to-end throughput measure the same thing?
Show answer and example
Answer: The analytical model estimates compute, hierarchical memory, pipelines, and idle tail work. End-to-end results include actual workload and system effects. Neither constants nor local optima transfer automatically.
Illustrative example: A faster GEMM tile may be offset by synchronization or other stages.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Can one A100 or A30 reproduce the MI350X/B200 production system?
Show answer and example
Answer: No. The original uses matching clusters and private workloads. This is a literature and cost review, not a reproduction. A public operator would need its own hardware and acceptance protocol.
Illustrative example: A small A100 GEMM cannot verify another cluster's end-to-end training throughput.
Sections in the Chinese source: 本地复现, 复现边界. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why predict both next and abstract latents?
Hierarchical Latent Prediction for Language Models
Show answer and example
Answer: Local rollout may accumulate error; coarse targets constrain longer-term structure while preserving immediate detail.
Illustrative example: Predict the next word state with an auxiliary constraint for the next passage.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why add a coarse temporal target to stepwise latent prediction in HiLP?
Hierarchical Latent Prediction for Language Models
Show answer and example
Answer: Small one-step errors can compound and lose distant structure. A coarse abstract latent supervises longer spans alongside next-token and local latent objectives. This does not eliminate rollout errors or replace long-horizon tests.
Illustrative example: Predict the next waypoint and the next stage to constrain gradual route drift.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
How does coarse latent loss differ from raising the next-step weight?
Hierarchical Latent Prediction for Language Models
Show answer and example
Answer: Reweighting supervises the same horizon. A coarse target introduces a different span and structure, not simply larger λ1.
Illustrative example: Doubling next-second loss does not ask for a prediction of the next minute.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
How isolate the effect of HiLP's coarse target?
Hierarchical Latent Prediction for Language Models
Show answer and example
Answer: Hold split, budget, and backbone fixed; ablate the block-level abstract objective and report fine and coarse errors. Synthetic diagnostics are not paper-scale LM results.
Illustrative example: Changing data and steps while adding λH confounds attribution.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why preserve dense forward but compress backward?
LazyFlash: Forward-Dense, Backward-Sparse Attention for Accelerated Long-Context Fine-Tuning
Show answer and example
Answer: Forward determines visible context. Uneven gradient contributions let training sample/reweight backward blocks without making inference forward sparse.
Illustrative example: Read the whole document, then sample only some backward blocks.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why does LazyFlash save backward work rather than visibility?
LazyFlash: Forward-Dense, Backward-Sparse Attention for Accelerated Long-Context Fine-Tuning
Show answer and example
Answer: Forward stays dense. Output-gradient block norms set backward retention probabilities, and inverse-probability weighting estimates dense gradients. A tiled backward kernel must skip actual work; masking afterward does not. A single random draw need not equal dense gradients.
Illustrative example: A low-contribution query block can be skipped in backward even though forward reads all evidence.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
Is importance backward equivalent to fixed top-k gradient truncation?
LazyFlash: Forward-Dense, Backward-Sparse Attention for Accelerated Long-Context Fine-Tuning
Show answer and example
Answer: No. Inverse-probability reweighting targets an unbiased dense-gradient estimate in expectation; fixed truncation generally does not. Individual draws still differ.
Illustrative example: A block kept with probability 0.5 is upweighted when sampled, not permanently dropped.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Why isn't masking gradients after dense backward a LazyFlash kernel?
LazyFlash: Forward-Dense, Backward-Sparse Attention for Accelerated Long-Context Fine-Tuning
Show answer and example
Answer: Compute and memory traffic have already occurred. Skip tiles inside FlashAttention backward. No local operator or GPU receipt is claimed.
Illustrative example: Discarding half the results after computing all blocks saves no backward execution.
Sections in the Chinese source: 背景与主要改动, 本地复现. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why store exponents separately from sign and mantissa?
LCT: Losslessly Compressed Tensors with GPU friendly execution for Training
Show answer and example
Answer: Exponents are often concentrated and compressible. Preserve sign/fraction verbatim, use a fixed bounded code, and handle runtime center shifts without losing bits.
Illustrative example: Shared exponent patterns can compress while fraction bits remain intact.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why retain rare exponents and overflow rather than drop them in LCT?
LCT: Losslessly Compressed Tensors with GPU friendly execution for Training
Show answer and example
Answer: Bit-exact recovery requires escape codes with raw rare exponents and raw overflow after stream exhaustion. Sign/fraction remain separate and a fixed codebook avoids relearning. This is lossless compression, not quantization.
Illustrative example: A rare exponent uses escape plus eight original bits; overflow stays raw rather than becoming zero.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
How does bounded exponent coding differ from zlib on a tensor?
LCT: Losslessly Compressed Tensors with GPU friendly execution for Training
Show answer and example
Answer: LCT specifies a fixed table, length cap, escapes, stream budget, and raw overflow tail. General compression does not provide that layout. Count codebooks and metadata in total bytes.
Illustrative example: Smaller payload may still produce larger total output after metadata.
Sections in the Chinese source: 背景与主要改动, 结果与独立对照. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Why compare raw bits in LCT round trips?
LCT: Losslessly Compressed Tensors with GPU friendly execution for Training
Show answer and example
Answer: Preserve signed zeros, subnormals, infinities, and NaN payloads. Numeric equality can miss these. Test all exponents, centers, and overflow paths.
Illustrative example: +0 and −0 compare equal numerically but have different bits.
Sections in the Chinese source: 核心公式, 数据协议与运行. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why combine trajectories, prototypes, and text?
Mobility-Embedded POIs: Learning What A Place Is and How It Is Used from Human Movement
Show answer and example
Answer: Sparse locations lack visits. Short trajectories connect to long-term prototypes, high-frequency anchors transfer temporal patterns, and text supplies semantic alignment.
Illustrative example: A rarely visited place can borrow time-of-day patterns from nearby busy places.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
How does ME-POIs transfer time information to sparse places?
Mobility-Embedded POIs: Learning What A Place Is and How It Is Used from Human Movement
Show answer and example
Answer: High-frequency anchors contribute within each spatial scale using distance normalization; then average scales so more anchors do not dominate. Trajectories connect recent visits and long-term prototypes; text supplies distinct semantic supervision.
Illustrative example: Normalize two local anchors and 20 broader anchors separately before averaging their predictions.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
How does per-scale normalization differ from one softmax over anchors?
Mobility-Embedded POIs: Learning What A Place Is and How It Is Used from Human Movement
Show answer and example
Answer: Normalize each scale independently before averaging. A pooled denominator changes relative scale weights based on candidate mass or count.
Illustrative example: Near and far scales each contribute one prediction regardless of candidate count.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Should repeated visits duplicate a place prototype as a negative?
Mobility-Embedded POIs: Learning What A Place Is and How It Is Used from Human Movement
Show answer and example
Answer: No. Eq. 9 uses distinct minibatch place prototypes. Keep temporal KL in direction r||q and sum across places as specified.
Illustrative example: Three visits to a POI still supply one prototype.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why optimize worst-domain performance instead of average alone?
A Theoretical Framework for Modular Learning of Robust Generative Models
Show answer and example
Answer: A good mean can hide severe regressions. Adversarial domain weights adjust input-dependent gates while respecting global probability mass.
Illustrative example: Two strong domains can conceal one failing domain in the average.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why doesn't per-input gate softmax necessarily normalize a generative mixture?
A Theoretical Framework for Modular Learning of Robust Generative Models
Show answer and example
Answer: Input-dependent weights cannot leave the global sum like fixed mixture weights. Local unit sums do not guarantee global mass one. Enforce total mass and distinguish worst-domain KL and NLL; silent post-normalization changes the objective.
Illustrative example: A gate can choose the highest-density expert separately at A and B, making their total mass exceed one.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
Are domain KL and NLL equivalent for gate and adversary updates?
A Theoretical Framework for Modular Learning of Robust Generative Models
Show answer and example
Answer: Within one domain they differ by gate-independent entropy, so gate gradients agree. Across unequal entropies the adversary's ordering can differ. Local practice uses NLL.
Illustrative example: Subtracting different domain entropies can change which domain is worst.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Why can't cached noncausal teacher targets enter the router input?
A Theoretical Framework for Modular Learning of Robust Generative Models
Show answer and example
Answer: Use them only as training targets. Causal router logits depend on prefix alone; the API does not prove the caller's network is causal.
Illustrative example: Knowing the target's expert probabilities during training does not let inference inspect the target token.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why is Muon a matrix geometry update?
Muon is Scalable for LLM Training
Show answer and example
Answer: Hidden weights form linear maps. Approximate polar decomposition acts on momentum jointly and uses shape scaling and decay.
Illustrative example: A dominant correlated direction is handled differently from independent scalar learning rates.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why treat gradients as matrices rather than adjust coordinates like AdamW?
Muon is Scalable for LLM Training
Show answer and example
Answer: Muon approximates the polar direction of momentum for hidden matrices, then applies shape scaling and decay. This uses spectral relationships, not just elementwise moments. Non-hidden parameters still use AdamW in the large recipe.
Illustrative example: Nearly collinear columns can yield different directions under matrix polar transformation and coordinate scaling.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
How does local Muon differ from applying it to every parameter?
Muon is Scalable for LLM Training
Show answer and example
Answer: Hidden matrices use Muon; embeddings, norms, biases, heads, and other parameters use AdamW. The hybrid optimizer is part of the comparison.
Illustrative example: A one-dimensional norm parameter is not suitable for matrix Newton–Schulz.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Why divide Newton–Schulz input by its Frobenius norm?
Muon is Scalable for LLM Training
Show answer and example
Answer: Normalize iteration scale before the specified polynomial; handle zero or tiny norms and preserve shape scaling. Orthogonality alone does not establish optimization quality or runtime.
Illustrative example: Scaling a gradient uniformly should not radically change the iteration's numeric range.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why report data, elapsed time, and tuning costs separately?
When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark
Show answer and example
Answer: Per-step and per-second quality and the cost of finding parameters differ. Equal-data gains need not hold at equal time or search budget.
Illustrative example: Lower loss for equal tokens can still lose after one hour if steps are slower.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why doesn't lower Muon loss at equal steps prove superiority at equal compute or tuning cost?
When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark
Show answer and example
Answer: Step latency, throughput, and tuning can differ. Pair initialization, data order, and candidates; compare processed data, elapsed training time, and fixed tuning budget separately. Freeze selection before test.
Illustrative example: Twice-slower steps can win at 100 steps but lose at ten minutes; retuning for time confounds the comparison.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
Why isn't random-access regret on a fixed candidate pool prospective adaptive HPO?
When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark
Show answer and example
Answer: It studies random order within precomputed candidates. Adaptive HPO uses prior results to propose or choose new trials, with different information and time costs.
Illustrative example: Randomly inspect 23 existing settings versus inventing a 24th from feedback.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Why sample the same seed indices in paired bootstrap?
When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget Benchmark
Show answer and example
Answer: Paired trajectories share initialization and data order. Mismatching seeds loses correlation. Integrate AUC over common measurement support and freeze validation-selected settings for all budget axes.
Illustrative example: Sample seed 42 for both optimizers, not 42 for one and 44 for the other.
Sections in the Chinese source: 核心公式, 结果与独立对照. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
What instability does Musec target?
Musec: MomentUm SpEctral Clipping for Stable Muon-type Training
Show answer and example
Answer: Smoothly clip excessive momentum singular directions so they do not dominate Muon-type updates. Controlling spectrum alone does not guarantee better loss.
Illustrative example: Limit a few huge singular values without equally scaling normal ones.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
How does Musec spectral clipping differ from shrinking all gradients?
Musec: MomentUm SpEctral Clipping for Stable Muon-type Training
Show answer and example
Answer: Smoothly constrain large momentum singular values without uniformly shrinking every direction. This changes update geometry, not proof of stability or final quality.
Illustrative example: Control one abnormally large direction without multiplying every normal direction by 0.1.
Sections in the Chinese source: 背景与主要改动, 核心公式或操作. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
What differs between smooth spectral clipping and a hard threshold?
Musec: MomentUm SpEctral Clipping for Stable Muon-type Training
Show answer and example
Answer: Hard decisions can jump at the boundary; smooth mapping responds continuously. Specify the exact rule and budget rather than just claiming stability.
Illustrative example: Slightly crossing a threshold need not abruptly change update magnitude.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Does local full-SVD soft clipping establish large-model throughput?
Musec: MomentUm SpEctral Clipping for Stable Muon-type Training
Show answer and example
Answer: No. It validates decomposition, smooth mapping, and reconstruction, not large training or optimized kernels. Verify numerical correctness and runtime separately.
Illustrative example: Small-matrix parity does not establish cost per step on a large model.
Sections in the Chinese source: 本地复现, 复现边界. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why audit states inside the training loop?
OPEN-1B: A Fully Auditable Training Run
Show answer and example
Answer: The final checkpoint does not explain its trajectory. Fix data and execution order and record hashes to replay and compare a specific step.
Illustrative example: If drift starts at step 100, inspect its inputs and state instead of only final accuracy.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why is a seed insufficient for auditing stepwise replay?
OPEN-1B: A Fully Auditable Training Run
Show answer and example
Answer: Batch order, kernel reductions, and collectives affect floating-point trajectories. OPEN-1B records them and per-step state hashes to localize discrepancies. Hashes detect but do not explain differences or guarantee cross-hardware parity.
Illustrative example: Different summation order can cause drift; hashes locate the first differing step.
Sections in the Chinese source: 背景与主要改动, 核心公式与实现对应. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
How does fixing seeds differ from fixing reductions, batches, and communication?
OPEN-1B: A Fully Auditable Training Run
Show answer and example
Answer: Seeds control only part of randomness. Floating reductions, batch order, and collectives must be explicit for replay; equal seeds do not imply bitwise equivalence across hardware.
Illustrative example: Changing addition order can introduce drift despite identical initialization.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Does matching a canonical hash prove full 400B-token reproduction?
OPEN-1B: A Fully Auditable Training Run
Show answer and example
Answer: No. Local work checks fixed-order reduction and hashing. Full audit requires actual batches, states, and replay; a hash neither executes training nor proves correctness.
Illustrative example: Equal hashes on two small tensors do not replay the paper's entire training.
Sections in the Chinese source: 本地复现, 复现边界. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why keep components as ordinary PyTorch modules?
Show answer and example
Answer: Tight coupling to training and distribution obscures structure and ablations. Plain modules and composable wiring improve transparency and portability across notebooks, CPUs, and GPUs.
Illustrative example: Replace one block without also replacing the training framework.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
How do composable Block, Residual, and Repeat help isolate ablations?
Show answer and example
Answer: Separate ordinary PyTorch modules from wiring such as repetition, residuals, and parallel branches. This reduces coupling to the trainer and distributed execution and aids matched comparisons. Composability is an engineering property, not quality evidence.
Illustrative example: Change serial to parallel wiring without changing data and training loop.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
Do DSL equivalence tests establish a better architecture?
Show answer and example
Answer: No. Equivalence checks preserve semantics under rewiring; architecture studies test quality after changes. Near-identical PPL supports equivalence, not improvement.
Illustrative example: Rewriting x+f(x) as Residual(f) should give the same result.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
How avoid incorrect Residual or Parallel wiring?
Show answer and example
Answer: Residual(f)(x)=x+f(x); Parallel(f,g)(x)=f(x)+g(x). Check dimensions and repetition parameter sharing. Parallel addition is not serial composition, and not all upstream presets are local.
Illustrative example: f(x)+g(x) generally differs from g(f(x)) at the same shape.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why tie example selection to task and budget?
PPL-Factory: Task-Aware and Budget-Aware Data Selection from Language Modeling to Reasoning
Show answer and example
Answer: No fixed easiest/hardest policy suits all budgets. Task-specific frozen-model NLL guides easy, middle, or mid-random selection.
Illustrative example: A narrow band at tiny budget may hurt coverage; random sampling within the middle pool offers another trade-off.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why does PPL-Factory vary the selection band by task and budget instead of always choosing hardest?
PPL-Factory: Task-Aware and Budget-Aware Data Selection from Language Modeling to Reasoning
Show answer and example
Answer: Difficulty and usefulness are not monotone. A frozen model scores under the task protocol, and easy, middle, or random middle sampling trade learning signal and coverage. Reasoning SFT ranks response loss only.
Illustrative example: With dozens of examples, hardest-only can lack useful signal; random midrange may retain coverage, while larger budgets differ.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
Can response-only and prompt-plus-response NLL ranks differ?
PPL-Factory: Task-Aware and Budget-Aware Data Selection from Language Modeling to Reasoning
Show answer and example
Answer: Yes. Full sequence includes prompt length and predictability; response-only isolates reasoning/answer tokens. Packed-block LM scoring is another protocol.
Illustrative example: A long easy prompt can hide a difficult answer in aggregate NLL.
Sections in the Chinese source: 背景与主要改动, 复现边界. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Why match frozen scorer and starting checkpoint across subsets?
PPL-Factory: Task-Aware and Budget-Aware Data Selection from Language Modeling to Reasoning
Show answer and example
Answer: Changing score model or initialization confounds selection. Match block count and tuning budget; preserve negative transfer of middle selection and do not retune on test.
Illustrative example: Random, Easy, and Middle must start from the same checkpoint to attribute PPL differences.
Sections in the Chinese source: 本地复现. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why care which states the student visits?
Reverse-KL Policy Distillation: Statistical Guarantees and Limits
Show answer and example
Answer: Accumulate teacher/student KL under candidate-specific visitation. Equal weighting of all historical states can misrepresent actual risk.
Illustrative example: A branch nearly never visited by a candidate should not count equally merely because it occurs in old rollouts.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why not fit token reverse KL on old rollout states alone?
Reverse-KL Policy Distillation: Statistical Guarantees and Limits
Show answer and example
Answer: Risk is under the candidate's visitation distribution. The method uses clipped prefix importance ratios and cumulative KL increments for bounded candidate comparison, then updates whole-policy weights. Exclude current action and weight both sides of the difference identically.
Illustrative example: A failure branch often visited by the old policy can be rare for the candidate and should not dominate.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
How does full-feedback teacher information differ from a sampled answer?
Reverse-KL Policy Distillation: Statistical Guarantees and Limits
Show answer and example
Answer: Complete action probabilities enable reverse KL and candidate comparisons at visited states. A single sample does not give them. Teacher support must include candidate-positive actions or KL can be infinite.
Illustrative example: Seeing A sampled reveals neither B nor C probabilities.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Why exclude the current action and apply the same prefix ratio to both terms?
Reverse-KL Policy Distillation: Statistical Guarantees and Limits
Show answer and example
Answer: r_h multiplies actions only at u<h and weights the difference of candidate and rollout cumulative KL increments. Including the current action or weighting one side changes the bounded objective.
Illustrative example: At state three, the arrival ratio uses only actions one and two.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why not keep only the low-rank projected gradient?
Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon
Show answer and example
Answer: Dominant singular directions can contain noise and exclude useful descent. Low-dimensional momentum handles in-subspace gradients; current orthogonal residual restores outside directions without full residual momentum.
Illustrative example: A smaller useful direction should not be permanently excluded by a large principal direction.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
How do SMRC and PMRC recover gradient directions outside low-rank momentum?
Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon
Show answer and example
Answer: Use low-dimensional momentum inside the subspace and current raw orthogonal residual outside, without persistent full residual state. SMRC uses low-dimensional EMA; PMRC modifies matrix direction treatment, and basis changes differ. Neither is just projected SGD.
Illustrative example: Keep momentum for two dominant directions while adding useful current gradient from a third.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
Why aren't SMRC and PMRC the same optimizer?
Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon
Show answer and example
Answer: SMRC maps EMA momentum back directly; PMRC first applies matrix sign. SMRC corrects parameters at stage boundaries, unlike the theoretical PMRC shift rule. One projected SGD function cannot substitute.
Illustrative example: Applying msign to M before mapping generally changes direction.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Why correct SMRC parameters and rerun backward before refreshing the basis?
Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon
Show answer and example
Answer: At stage end, correct parameters and zero momentum. A new forward/backward at the changed parameters supplies the refresh gradient. A stale gradient mismatches state; local β weights the new gradient, unlike some APIs.
Illustrative example: Do not reuse the gradient from before the parameter correction.
Sections in the Chinese source: 核心公式, 已实现的组件. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why put related matrices on the third tensor axis?
T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training
Show answer and example
Answer: It lets layers, QKV, or heads interact through a third-axis transform before slice-wise polar and inverse mapping. Both grouping and transform define the method.
Illustrative example: Stacking Q/K/V couples them instead of handling three unrelated matrices.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why stack related matrices into a tensor instead of flattening them for ordinary Muon?
T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training
Show answer and example
Answer: Transform along the third axis, compute polar directions slice-wise, and invert the transform. This couples related matrices. Grouping and transform are separate choices. Restore original matrix scaling and zero rank-zero slices to avoid spurious updates.
Illustrative example: Q/K/V transform-domain slices are not equivalent to one flattened matrix.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
Why ablate grouping and transforms separately?
T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training
Show answer and example
Answer: Grouping decides which matrices share a tensor; the transform decides how slices mix. Changing both confounds causes.
Illustrative example: QKV with DCT versus heads with DFT does not isolate DFT.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Should fan scaling use head slices or original matrices?
T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training
Show answer and example
Answer: Use original fan_out/fan_in after reconstruction. Rank-zero slices must yield zero instead of arbitrary thin-SVD nullspace updates. Exact local SVD does not establish production polar throughput.
Illustrative example: Even after slicing by head, scale using the unsplit dimensions.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why keep one position per column?
TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning
Show answer and example
Answer: The max absolute entry yields a one-sparse signed direction and permits low-precision column-level state instead of dense per-parameter state.
Illustrative example: A column with many nonzeros updates only its largest-magnitude entry this step.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
How is one gradient position per TACO column different from elementwise sign?
TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning
Show answer and example
Answer: Select the maximum absolute gradient row per column and retain only its sign. Low-precision column state saves memory. SignSGD updates all nonzero locations and is not one-sparse.
Illustrative example: [0.2,−0.9,0.4] yields only a negative update at row two rather than +,−,+.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
What differs between TACO and elementwise signSGD?
TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning
Show answer and example
Answer: TACO chooses one max-absolute row and keeps its sign; signSGD can update all nonzero entries. Their supports differ even though both use signs.
Illustrative example: For [1,−4,2], only TACO's second entry is −1 and the others are zero.
Sections in the Chinese source: 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
What if TACO's argmax runs along the wrong axis?
TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning
Show answer and example
Answer: Maximize over rows separately for each column, not columns per row. Check zeroed positions and low-precision column momentum, not shape alone.
Illustrative example: A 3×5 matrix has up to one selected entry in each of five columns.
Sections in the Chinese source: 核心公式, 复现边界. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Basics
Why make training-example value a learned prediction?
Show answer and example
Answer: Teacher loss gaps create pseudolabels, then a selector scores an independent pool. This amortizes expensive valuation rather than rerunning training across the large pool.
Illustrative example: Estimate value on a small set, then score candidates unseen by the teacher models.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Core mechanism
Why train TESS on loss-gap pseudolabels instead of directly optimizing selection?
Show answer and example
Answer: Compare train-only and validation-guided models from the same initialization per example, then learn a transferable pointwise selector. Separating valuation from prediction avoids some degenerate direct objectives. Validation guidance is not test data.
Illustrative example: Use loss changes to learn which examples help, then predict candidate value without reading test outcomes.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Method comparison
How do loss gaps differ from selecting the highest raw loss?
Show answer and example
Answer: A shared-init model pair estimates response to validation guidance; raw loss measures difficulty. Hard examples need not be useful.
Illustrative example: Persistent noise can have high loss without a valuable guided change.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Foundation models / Training budgets: data mixtures, objectives, and optimizers
Implementation details
Can selector gradients flow back through Δ_i into U/W?
Show answer and example
Answer: No. Train U/W in order from identical initialization and detach their saved loss gaps. Scores may be negative; do not use them as probabilities without transformation or substitute test for validation.
Illustrative example: A negative value label should not be silently clamped to zero.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Topic guide and reading order
Start with this question¶
Under a limited budget, should we change the data, the update direction, or the model size first?
Takeaway¶
DoReMi learns a domain mixture with Group DRO using a small proxy model's per-domain excess loss over a reference, then transfers the mixture to target training. Muon approximately orthogonalizes momentum updates for hidden matrices with decay and shape scaling. Other parameter types cannot blindly use the same update. The first changes what the model sees; the second changes how updates move. Evaluate the axes separately.
Example¶
A one-third split across code, web, and encyclopedia data is only a starting point. DoReMi uses difficulty relative to a reference, not the domains with the highest raw loss. After switching to Muon, compare matched-token, time, or FLOP curves, not one matched-step point.
Common pitfall¶
A proxy's mixture is not guaranteed optimal at every scale; optimizer gains depend on learning rate and budget. Low-precision states, activation compression, and data selection can affect different resources. Retain the documented limits of mechanism checks, short-budget negative runs, and costly unreproduced experiments.
Reading order¶
Start with why DoReMi uses excess rather than raw loss. Then study Muon's matrix update and limits for non-matrix parameters. Move on to data selection, low-precision states, and training systems.
- DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining · Original paper
- Muon is Scalable for LLM Training · Original paper
Check your understanding¶
Optimizer A lowers loss faster per step but takes twice as long per step as B. Does it use less compute?
Show suggested answer
Not necessarily. Decide whether the objective is time, FLOPs, or tokens to matched quality, then compare the relevant curves. Match training data and tuning budget. Per-step progress is not end-to-end efficiency.