Multimodal models¶
简体中文 · English
Study three questions: how to align modalities, how to budget visual evidence, and how to predict states and the consequences of actions.
Choose a subtopic and start with its paper questions.
Cross-modal alignment: representations, connectors, and supervision
How do images, speech, and text connect to the same task? Does the model learn similarity or learn to generate answers?
Visual evidence budgets: tokens, frames, and on-demand revisiting
When visual input is too long, which details can be skipped for now, and which need a way to be recovered?
Visual world models: state, generation, and action conditioning
Does the model only recognize a scene, or can it predict how an action changes the state?