LLM post-training¶
简体中文 · English
Start by identifying the feedback source, then study policy updates, credit assignment, teacher supervision, and data and compute allocation.
Choose a subtopic and start with its paper questions.
Preference and alignment: identify the feedback first
How do we turn pairwise preferences or a thumbs-up on one answer into a trainable objective?
Policy optimization: which policy samples and which policy learns?
Which quantities must remain distinct across online sampling, old-policy probabilities, and current-policy updates?
Reward and credit: which steps deserve a good final score?
How can one final score become a training signal that does not reward wrong steps?
Trajectory distillation: students, proposals, and prefix distributions
If a student generates from its own mistaken prefixes at inference, why might imitating complete teacher answers miss those states?
Teachers and correction: when extra context can help
How can a teacher with answers, visual evidence, or specialist hints help a student that cannot see them?
Data and budgets: where to spend the next unit of compute
Should a limited budget buy more questions, more rollouts, or more supervision at valuable positions?