Foundation models¶
简体中文 · English
Understand context access, allocation of compute and training budgets, and trade-offs in compression, generation, and evaluation.
Choose a subtopic and start with its paper questions.
Long context: attention, recurrent state, and memory
As context grows, does the model preserve addressable historical tokens or a compressed recurrent state?
Conditional computation: experts, elastic depth and width, and lookup memory
At a fixed parameter budget, which parameters and how many steps should each token activate?
Training budgets: data mixtures, objectives, and optimizers
Under a limited budget, should we change the data, the update direction, or the model size first?
Inference compression: weights, KV cache, and serving bandwidth
When memory or latency is constrained, do weights, historical states, or data movement dominate?
Generation paths: draft verification, parallel denoising, and few-step refinement
How can we reduce serial token generation while being explicit about whether the output distribution changes?
Capability evidence: sampling, readout, and reliability
How can we tell whether a high score reflects the intended capability rather than language, ordering, readout, or statistical bias?