Visual world models: state, generation, and action conditioning¶
简体中文 · English
Multimodal models / 96 questions · 24 papers
Try answering each question before opening the explanation. The bank grows as papers and questions are added.
Read this topic's comparisons, analysis, and cross-paper Q&A
96 questions · 24 papers
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why does ALPS want one visual representation for language, state prediction, and image inspection?
ALPS: Visual State Encoders for World Models
Show answer and example
Answer: Planning needs predictability, instruction tasks need language alignment, and inspection needs readable image recovery. Shared patch tokens support those interfaces, but each must be tested. Pixel reconstruction alone does not establish planning, nor language alignment complete physical state.
Illustrative example: A robot needs to understand “move the cup right,” predict its location, and inspect the future image.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why supervise only teacher-crop B patches that fall inside student crop A in stage two?
ALPS: Visual State Encoders for World Models
Show answer and example
Answer: The student sees only A, so exact targets outside A would supervise invisible content. Map geometry first, then predict frozen teacher targets at corresponding positions; appearance augmentation promotes robustness. Shared initialization does not mean the teacher updates with the student.
Illustrative example: If the student sees the left half, supervise only the overlap, not small text on the unseen right.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
How does the fixed stage-one teacher differ from an EMA teacher?
ALPS: Visual State Encoders for World Models
Show answer and example
Answer: Stage one initializes the student and a teacher frozen through stage two. EMA would move targets with student weights, which is a different method. Training drops the cross-view predictor afterward and deploys the student encoder and language interface.
Illustrative example: Student augmentation training may change the encoder while fixed clean-crop teacher targets remain unchanged.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
Why average the cross-crop loss over valid patches per image before averaging the batch?
ALPS: Visual State Encoders for World Models
Show answer and example
Answer: Overlap sizes differ; per-image means prevent large overlaps dominating. A B patch is valid if its center falls in A's half-open bounds. Empty overlap safely contributes zero, while the sampler separately enforces a minimum valid count.
Illustrative example: With 100 matching patches in one image and ten in another, a flat token mean gives tenfold weight to the first; image means do not.
Sections in the Chinese source: 核心公式, 本地复现, 验证与结果边界. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why combine composable programs, future vision, and robot action in one AW-0 model?
Show answer and example
Answer: Programs provide reusable steps, video anticipates scenes, and continuous actions control the robot. Three experts jointly link plan to physical consequences; generated code plans by default rather than being executed. Local work reviews evidence only.
Illustrative example: Generate “grasp then place,” forecast the scene, and produce controls without running arbitrary Python.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why allow bidirectionality within AW-0 experts but enforce code→video→action across them?
Show answer and example
Answer: Bidirectionality organizes representations inside a modality; block causality preserves conditioning from plan to scene prediction to control. Code must not see later videos or actions, while action can use earlier code and predicted scenes. Full mutual visibility changes inference order and available information.
Illustrative example: Generate a cup-grasp program, predict the scene, then output controls; do not give the planner future ground-truth actions.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
How do separate expert projections, normalization, and FFNs differ from fully shared transforms?
Show answer and example
Answer: Separate transforms serve code, video, and action while joint attention exchanges information at prescribed boundaries. They add parameters and interfaces, not guaranteed improvement. The paper combines bidirectional intra-expert and block-causal cross-expert attention.
Illustrative example: Discrete code and continuous actions can use separate FFNs and still interact, without being isolated or freely bidirectional.
Sections in the Chinese source: 背景与主要改动. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
Can flow targets and masked code diffusion use one noise and supervision mask?
Show answer and example
Answer: No. Video and action use continuous noising and velocity regression. Code uses an independent Bernoulli mask and loss only on masked program tokens. Action noise and code masking are not determined by video noise. Mixed video sampling also needs separate clipped and normalized importance weights.
Illustrative example: Low video noise does not imply lightly masked code; one shared mask changes the paper's independent sampling.
Sections in the Chinese source: 本地收录与复现边界. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why might a Decision Transformer need more than expected return?
DT-JEPA: Learning to Predict Future Representations for Decision-Making
Show answer and example
Answer: Equal returns can lead to different states, so one return-to-go cannot distinguish dynamics. DT-JEPA conditions actions on a predicted future representation distribution from history, not an observed future. Manage uncertainty and isolate training-only inputs.
Illustrative example: Routes with equal reward may reach a charger or an exit; state predictions provide information absent from the scalar.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why predict future latents from history and expose true future only to the training target encoder?
DT-JEPA: Learning to Predict Future Representations for Decision-Making
Show answer and example
Answer: The future is unavailable at decision time, so reading it would leak. History predicts a distribution and reparameterized samples condition decisions. The EMA target only supplies training alignment. Future representations distinguish dynamics even at equal return.
Illustrative example: Two trajectories can score ten, yet one hits an obstacle and the other stays clear.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
What is the trade-off between a Gaussian future distribution and a deterministic vector?
DT-JEPA: Learning to Predict Future Representations for Decision-Making
Show answer and example
Answer: Mean and variance express uncertainty; reparameterized samples condition DT. A deterministic vector is a single point. The diagonal Gaussian is still limited and does not cover arbitrary multimodality or allow real future input at deployment.
Illustrative example: Variance can represent uncertainty between routes, but a single Gaussian may not capture two distant discrete destinations.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
Can NLL, BC, and critic reductions all use generic MSE.mean?
DT-JEPA: Learning to Predict Future Representations for Decision-Making
Show answer and example
Answer: No. Sum NLL over latent dimensions before the batch mean; sum BC and critic over time and action dimensions before the batch mean. Reductions change relative weights. Detach target and TD bootstrap, but retain actor gradients through Q with respect to actions. Local tests omit full online data collection.
Illustrative example: Doubling action width changes an all-element mean's per-example contribution relative to other summed terms.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why improve both question planning and perceptual answers in egocentric spatial reasoning?
EgoP²C: Internalizing Plan-Perceive-Compose for Spatial Reasoning in Egocentric Video
Show answer and example
Answer: Complex questions need decisions about where to look and what to ask, followed by reliable evidence extraction. Errors in either role propagate. EgoP²C improves both while respecting their information boundaries; final conclusions still need validation.
Illustrative example: For a drawer relative to a door, plan a viewpoint and subquestion; a left/right perceptual error cannot be fixed by fluent composition.
Sections in the Chinese source: 背景与主要改动, 核心公式与信息边界. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why must role teachers score the same student tokens in segmented distillation?
EgoP²C: Internalizing Plan-Perceive-Compose for Spatial Reasoning in Egocentric Video
Show answer and example
Answer: Actual student interaction defines visited prefixes. Role-specific teachers score those prefixes to identify planning, perception, or direct-answer errors. A separate teacher-written answer loses on-policy supervision. Shared video KV must exclude role-private history.
Illustrative example: If a student asks for a viewpoint then makes a spatial judgment, teachers score the corresponding segments of that same trace.
Sections in the Chinese source: 背景与主要改动, 核心公式与信息边界. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
Do the PC, V, and general G experts share the same parameter training boundaries?
EgoP²C: Internalizing Plan-Perceive-Compose for Spatial Reasoning in Egocentric Video
Show answer and example
Answer: Experts initialize from copies of the base FFN. G's FFN stays frozen, while its attention residual trains only in MOPD. PC and V supply trainable planning and perception paths. This preserves general answering without updating three full models alike.
Illustrative example: Keeping G's FFN does not forbid all adaptation; its attention residual can train later.
Sections in the Chinese source: 背景与主要改动, 核心公式与信息边界. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
Why normalize role length over the whole trajectory?
EgoP²C: Internalizing Plan-Perceive-Compose for Spatial Reasoning in Egocentric Video
Show answer and example
Answer: A role may have multiple segments; divide by all supervised tokens for that role in the trajectory, not each segment separately. Otherwise splitting a segment increases role weight. Detach teacher and old-student log probabilities, keep private decoder KV separate, and do not interpret random-teacher diagnostics as spatial capability.
Illustrative example: Ten PC tokens in two five-token segments should keep the same total weight. Separate means would double their contribution.
Sections in the Chinese source: 核心公式与信息边界, 本地复现. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why learn from human egocentric videos for transferable robot experience?
EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos
Show answer and example
Answer: Robot demonstrations are costly and embodiment-specific. Human video has action–appearance relationships but not robot control signals. Learn latent visual and action structure before adapting to a robot; actual control still needs environment validation.
Illustrative example: Human drawer-opening video teaches scene change, not direct robot-arm joint commands.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why learn latent actions instead of mapping human hand trajectories directly to robot joints?
EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos
Show answer and example
Answer: Human hands and robot actuators have different geometry. Motion-difference conditioning learns latent actions linked to visual changes, then joint modeling and robot-specific interfaces adapt them. What transfers is the effect, not coordinate equivalence.
Illustrative example: Fingers pinching and a gripper closing have different angles but can both lift a cup.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
How do action-conditioned JEPA and later joint visual/latent-action flow matching differ?
EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos
Show answer and example
Answer: First predict future visual representations conditioned on hand-motion differences, linking latent actions to observable effects. Then jointly model visual and action latents and adapt a robot I/O interface. Prediction and joint generation are separate and do not by themselves establish robot control.
Illustrative example: Learn which hand changes produce which visual effects before adapting the learned joint structure to an arm.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
How does visibility of future visual targets differ between JEPA training and control?
EgoJEPA: Beyond Retargeting for Dexterous Manipulation from Egocentric Videos
Show answer and example
Answer: In training, an encoder supplies detached targets from future frames for prediction from history and hand-motion differences. Control cannot read an unrealized future. Local work is a literature/cost record with no adapter, metrics, or GPU receipt.
Illustrative example: Use the later object-position frame to supervise prediction, not as a deployment input.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why does ESC care whether objects persist across generated viewpoints?
ESC: Escaping 2D Diffusion Ambiguity with Geometric Constraints for Consistent Novel View Synthesis
Show answer and example
Answer: Generators can render plausible posed images while adding or dropping objects. ESC links RGB and depth to test geometric consistency. New views should differ in pixels in spatially explainable ways, not merely look similar or good.
Illustrative example: A cup should move in projection as the camera circles a table, not vanish in an unoccluded region.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why can a sharp image with the right camera pose still fail PCC?
ESC: Escaping 2D Diffusion Ambiguity with Geometric Constraints for Consistent Novel View Synthesis
Show answer and example
Answer: Realistic appearance does not ensure the same 3D scene. PCC uses bidirectional depth reprojection and visibility to detect vanished or hallucinated objects, respecting occlusion and invalid pixels. Best-of-N selects geometric consistency, not the prettiest candidate.
Illustrative example: A cup can disappear in a visually clean view; bidirectional reprojection detects the inconsistency.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
How does PCC Best-of-N differ from selecting the closest RGB image?
ESC: Escaping 2D Diffusion Ambiguity with Geometric Constraints for Consistent Novel View Synthesis
Show answer and example
Answer: Correct viewpoints can change pixels. PCC projects source and target depths both ways with occlusion and valid coverage checks, detects missing or hallucinated geometry, and minimizes the score. This does not optimize every image-quality metric.
Illustrative example: A duplicate source image can have the smallest RGB difference while failing the requested camera view.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
Why do reprojection checks need z-buffering and valid-pixel masks?
ESC: Escaping 2D Diffusion Ambiguity with Geometric Constraints for Consistent Novel View Synthesis
Show answer and example
Answer: Multiple source points can land on one pixel and must respect occlusion. Negative depth, out-of-bounds projections, and missing source depth are not valid errors. PCC also uses thresholds, logarithmic compression, clipping, and coverage normalization. No local implementation or capability metrics.
Illustrative example: A wall can correctly occlude an object. Lack of a visible match is not disappearance.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why support phone, server, and split-device budgets for one video model?
EV-JEPA: Elastic V-JEPA for Any Device and Cross-Device Inference
Show answer and example
Answer: A fixed width may not fit all devices. Separate models may use incompatible representations. EV-JEPA learns shared elastic widths and compatible handoffs, but not every split is valid or free. Transfer, compression, and task quality need testing.
Illustrative example: Matching tensor shapes is not enough for a phone's narrow front half to hand off to a wide server back half.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why train narrow-to-wide as well as narrow and wide paths?
EV-JEPA: Elastic V-JEPA for Any Device and Cross-Device Inference
Show answer and example
Answer: Individually strong widths need not exchange intermediate states successfully. The mixed route trains the widening boundary under a common frozen teacher. This differs from arbitrary slicing or splicing independent models and does not eliminate communication cost.
Illustrative example: A server's wide layers may not understand a phone's narrow intermediate state without handoff training.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
How does a learned split-layer compressor differ from dropping tokens?
EV-JEPA: Elastic V-JEPA for Any Device and Cross-Device Inference
Show answer and example
Answer: Dropping tokens may permanently hide evidence. A compressor trains reconstruction so transmitted representations remain useful to the second stage. Lower reconstruction loss is not end-to-end losslessness or proof of deployed handoff.
Illustrative example: Dropping half the spatial positions differs from encoding into fewer tokens and learning to reconstruct recoverable structure.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
Does EV-JEPA's teacher stay frozen or track the elastic student by EMA?
EV-JEPA: Elastic V-JEPA for Any Device and Cross-Device Inference
Show answer and example
Answer: Frozen V-JEPA 2 supplies common targets for narrow, wide, and mixed student paths. Nested student parameters share masked-token L1 supervision in one space. Channel slicing alone is not joint training or cross-device verification.
Illustrative example: Updating shared student weights leaves the teacher's outputs fixed, not updated through EMA.
Sections in the Chinese source: 核心公式, 本地复现, 边界与后续触发条件. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why reuse video generation knowledge for 3D and 4D perception?
Gen4P: Video Generators are 4D Perceivers
Show answer and example
Answer: Generation pretraining can learn appearance, motion, and geometry priors relevant to depth, tracks, and cameras. Gen4P extracts a clean-video scene representation and queries properties. It still needs geometry targets or task validation, not new video generation.
Illustrative example: Given a moving-camera clip, query a point's 3D trajectory instead of asking for another clip.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why can a video generator support 4D perception without multistep denoising?
Gen4P: Video Generators are 4D Perceivers
Show answer and example
Answer: Gen4P reads motion and geometry representations inside a pretrained DiT. Clean video latents get one prescribed conditioned encoding; queries read tracks or depth. Frozen geometry adapters and trainable task LoRA differ. Supervision and decoders remain necessary.
Illustrative example: Train a head to query a point's 3D location after three seconds without generating another video.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
How do frozen depth-LoRA and trainable 4D-LoRA differ from full fine-tuning?
Gen4P: Video Generators are 4D Perceivers
Show answer and example
Answer: Freeze the base DiT and depth LoRA; train the 4D LoRA, queries, and decoders. This preserves generation and depth priors more directly than updating all weights. It does not guarantee every metric improves; the paper reports a depth regression.
Illustrative example: Train a temporal tracking adapter without modifying the depth update, then test both tasks.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
What distinctions matter for continuous time queries and DiT input?
Gen4P: Video Generators are 4D Perceivers
Show answer and example
Answer: Query time interpolates endpoint embeddings with t/(T−1). DiT reads clean z0 under the final scheduler time and an empty text condition. Query time is not diffusion noise time, and this is not iterative denoising. Local work is documentation only.
Illustrative example: A midpoint time query interpolates embeddings; it does not add half-strength noise to the frame.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why does Inside-JEPA want latent states to encode relationships?
Inside-JEPA: From Prediction to Relational Grounding in Latent World Models
Show answer and example
Answer: Latent forecasts need not expose contact or relative position. Inside-JEPA models entities and relations and studies how action supervision shapes them. A relation channel or low prediction loss does not establish learned relational semantics; controls are needed.
Illustrative example: Two unchanged-looking objects can transition from separated to touching, altering the right action despite unchanged categories.
Sections in the Chinese source: 背景与主要改动, 核心公式与五臂. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why use a differentiable online future encoding in the inverse-dynamics head instead of EMA?
Inside-JEPA: From Prediction to Relational Grounding in Latent World Models
Show answer and example
Answer: The inverse objective must send action-explanation gradients into relationship representations on the future side. JEPA prediction can still use a stable EMA target. Entity-only and shuffled-action controls test whether relations truly supply action information.
Illustrative example: Predict the action from changes in cup–box relations; the future relation encoder needs gradients to learn such changes.
Sections in the Chinese source: 背景与主要改动, 核心公式与五臂. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
What do INV-S, INV-R, and shuffled-label controls isolate?
Inside-JEPA: From Prediction to Relational Grounding in Latent World Models
Show answer and example
Answer: INV-S reads relations only for inverse dynamics; INV-R reads entity latents only. Shuffling changes only inverse labels, while prediction still reads true actions. These controls isolate supervision sources rather than treating any auxiliary gain as relation grounding.
Illustrative example: Similar gains with random action labels suggest regularization rather than learned action–relation correspondence.
Sections in the Chinese source: 背景与主要改动, 核心公式与五臂. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
Why is the four-entity K=8 state 256-dimensional, not 224 after removing diagonals?
Inside-JEPA: From Prediction to Relational Grounding in Latent World Models
Show answer and example
Answer: Entity latents occupy 4×32 and ordered pair slots occupy 4×4×8. Padding and diagonals stay present but zero. JEPA uses detached EMA future targets; inverse dynamics needs differentiable online future state. Mixing them cuts a defining gradient.
Illustrative example: Zeroing diagonal relationships does not delete their 32 slots or shrink the predictor interface.
Sections in the Chinese source: 核心公式与五臂, 本地复现与操作. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why does a small one-step JEPA latent loss not establish understanding of physical motion?
One-Step Prediction Fidelity Is a Poor Proxy for Planning Performance in JEPA World Models
Show answer and example
Answer: Latents use encoder-specific coordinates; closeness need not mean accurate position, angle, speed, or long-horizon planning. This work checks physical decoding and actual planning separately, with shared training scales.
Illustrative example: Similar latent vectors may decode to different object angles, so measure physical error and planning success.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why subtract decoding error on true next representations from decoding error on predicted ones?
One-Step Prediction Fidelity Is a Poor Proxy for Planning Performance in JEPA World Models
Show answer and example
Answer: Decoding error mixes encoder readability, decoder limitations, and prediction error. Using the same frozen decoder for both isolates excess error attributable to prediction more closely. It can be negative and must not be clipped. Long-horizon planning needs separate paired tests.
Illustrative example: If real representations decode two cells off and predicted ones three cells off, do not attribute all three cells to prediction.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
Why measure one-step physical error and planning success separately?
One-Step Prediction Fidelity Is a Poor Proxy for Planning Performance in JEPA World Models
Show answer and example
Answer: A planner repeatedly queries the model; errors can accumulate around key actions or alter search. One-step means miss closed-loop effects. Test actual planning with paired seeds and the same checkpoint and task.
Illustrative example: Better accuracy on common stationary states can lower average error even if near-collision mistakes make plans crash.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
Should negative E_excess be clipped to zero?
One-Step Prediction Fidelity Is a Poor Proxy for Planning Performance in JEPA World Models
Show answer and example
Answer: No. It is decoded error from predictions minus decoded error from true representations; predictions may happen to fit the frozen decoder better. Identical representations must yield zero. Fit state variance on shared training data, never refit the decoder on predicted test latents.
Illustrative example: True error three and predicted error two gives −1; clipping hides decoder extrapolation behavior.
Sections in the Chinese source: 核心公式, 本地复现, 聚合、结果与边界. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why link predicting consequences with producing usable actions in LeWAM training?
LeWAM: End-to-End World and Action Modeling with JEPAs
Show answer and example
Answer: A world model can predict an action's future without finding goal-reaching actions efficiently. A separate policy may not shape visual representations. Joint state, dynamics, and stochastic action learning shares signals, but does not remove isolation or execution requirements.
Illustrative example: Knowing a push moves an object is not the same as deciding which direction and distance to push.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why prevent LeWAM's noisy-action branch from seeing clean actions indirectly?
LeWAM: End-to-End World and Action Modeling with JEPAs
Show answer and example
Answer: Clean actions supervise future dynamics for state queries but are unavailable to the deployed action policy. Indirect access through shared attention would let the noisy branch copy targets. Separate parameters alone do not prevent leakage; enforce visibility across layers.
Illustrative example: A provided “push left” label can train prediction but must not be read by the action generator and counted as a decision.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
How does action flow differ from wide Gaussian random search?
LeWAM: End-to-End World and Action Modeling with JEPAs
Show answer and example
Answer: Flow matching learns velocities from noise to action data, proposing candidates nearer the learned distribution before planning in latent dynamics. Wide Gaussian search relies on unstructured coverage. Actual task advantage requires controlled validation.
Illustrative example: A learned proposal might favor plausible pushes along a table instead of many trajectories through it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
Why refine actions with a proximity penalty but select by the original goal cost?
LeWAM: End-to-End World and Action Modeling with JEPAs
Show answer and example
Answer: Adam penalizes deviations from the sampled plan to constrain refinement. Final selection compares unpenalized goal cost, retaining original samples in case refinement worsens them. Planning updates actions, not model weights, and does not guarantee real success.
Illustrative example: A refined action that deviates more can still win if it reaches the predicted goal more closely; keep the original if refinement fails.
Sections in the Chinese source: 核心公式, 本地复现, 数据协议、结果与边界. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why prevent collapse if two views of the same image should agree?
LocaLeJEPA: Maximum Entropy Representations under Geometric Priors
Show answer and example
Answer: A constant vector also gives perfect view agreement. LocaLeJEPA wants invariance across augmentations and distinguishable representations across inputs while respecting local geometry. A low agreement loss does not establish useful features.
Illustrative example: Cat crops should be close, but collapsing cats and cars together ruins retrieval.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why not pull all LocaLeJEPA clusters toward one Gaussian?
LocaLeJEPA: Maximum Entropy Representations under Geometric Priors
Show answer and example
Answer: Local clusters have different geometries and positions. A global pull can destroy layout. Fix group assignments, recenter residuals per cluster, and match a uniform solid-ball distribution, with an explicit small-cluster fallback.
Illustrative example: Two clusters for different motion modes can keep distinct centers while each has a regular local shape.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
How does a uniform solid ball differ from a Gaussian or a unit sphere surface?
LocaLeJEPA: Maximum Entropy Representations under Geometric Priors
Show answer and example
Answer: The solid ball puts mass inside radius √(d+2), giving identity covariance. A Gaussian is unbounded and a sphere surface has mass at a fixed radius only. Their characteristic functions differ. Gaussian formulas or vector normalization cannot replace the target.
Illustrative example: A uniform disk includes points near its center; a circle does not, and a Gaussian extends to arbitrary radius.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
How do small-cluster filtering and cluster count change regularization strength?
LocaLeJEPA: Maximum Entropy Representations under Geometric Priors
Show answer and example
Answer: Keep clusters with at least four samples; fall back to the full batch and log it if none qualify. Multiply each cluster statistic by sample count, then average equally over valid clusters and views. Count changes both granularity and scale. Integrate the ball characteristic function, not a Gaussian target.
Illustrative example: Splitting into more valid small clusters changes the value through weighting and averaging even with the same λ.
Sections in the Chinese source: 核心公式, 本地复现, 验证与边界. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
How does attribute leakage violate a multi-instance edit?
Editing Everything Everywhere All at Once
Show answer and example
Answer: Different objects may need different changes, but unbound color, shape, or texture can spread to other objects. MICE maintains local assignments and contextual blending without new training. Success is applying each requested change only to the right instance, not just making a prettier image.
Illustrative example: Left cup red and right cup blue should not yield two purple cups.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why need both text-group isolation and smooth spatial bias for multi-instance editing?
Editing Everything Everywhere All at Once
Show answer and example
Answer: Text isolation stops one instruction directly modifying another group's semantics. Smooth spatial bias binds edits to instances while blending at boundaries. Hard cuts can show seams; global text can leak color. This attention control does not train a new editor.
Illustrative example: Keep “left shoe red” and “right shoe blue” semantically separate while blending each localized edit naturally.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
How does smooth spatial bias differ from a hard mask outside the region?
Editing Everything Everywhere All at Once
Show answer and example
Answer: Log-smoothed masks give finite negative bias that fades near boundaries; hard masks prohibit interactions. Unrelated text groups still use negative infinity, and pixels belonging to other instances are zeroed. Local behavior follows paper_v1, not every author default.
Illustrative example: Weak interactions near a shadow can aid blending while isolating another instance's exclusive pixels.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
Why use at least float32 for ε=10^-30 and avoid arbitrary context-to-vision masks?
Editing Everything Everywhere All at Once
Show answer and example
Answer: FP16 can underflow ε and turn finite log bias into −∞. Softmax and accumulation need at least FP32. The local implementation follows text Case 5: context sees the full visual domain without spatial bias, rather than mixing incompatible diagram or processor defaults.
Illustrative example: Computing log(ε) after underflow changes intended soft decay into a hard exclusion.
Sections in the Chinese source: 核心公式, 本地复现, 原文与代码版本边界. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why might a small-batch SSL model align views without learning distinguishable features?
Dataset-Level Regularization for Joint-Embedding Self-Supervised Learning
Show answer and example
Answer: Positive alignment alone allows a constant encoder. Anti-collapse needs overall spread, which small batches estimate poorly. OOB uses broader statistics under memory constraints, but downstream performance still needs independent evaluation.
Illustrative example: Several similar embeddings may reflect genuinely similar images or global collapse; view agreement alone cannot tell.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
How does OOB provide near-dataset-scale anti-collapse supervision with a small batch?
Dataset-Level Regularization for Joint-Embedding Self-Supervised Learning
Show answer and example
Answer: Periodically encode the training set and cache anchors per sample, then read fixed targets during small-batch optimization. This avoids estimating the full distribution every step but requires refresh as the model changes. It is not a minibatch variance or covariance loss.
Illustrative example: If memory holds eight images, precompute full-training-set quantile targets and match each of those eight to its own anchor.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
How does Gaussian sorted-quantile matching differ from enforcing zero mean and unit variance?
Dataset-Level Regularization for Joint-Embedding Self-Supervised Learning
Show answer and example
Answer: Sorted quantiles assign normal targets to samples along random projections, constraining distributions and assignments. Mean and variance constrain only two moments. This is not the original Epps–Pulley statistic; matched baselines need the same quantile loss.
Illustrative example: Equal mass at −1 and +1 has mean zero and variance one but not normal quantiles.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
How should refresh directions, targets, and gradients be isolated?
Dataset-Level Regularization for Joint-Embedding Self-Supervised Learning
Show answer and example
Answer: Freeze refreshed directions and per-sample targets, detach them and store on CPU. Normalize online loss by batch B, views V, and directions M; no gradients through refresh. The spherical variant optimizes unit anchors, not Gaussian targets. Stable sample IDs are required.
Illustrative example: After dataloader shuffling, look up each original ID rather than pairing the first cached anchor with the first current image.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界与成本结论. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why do long video generators forget scene layout when returning to an old viewpoint?
Persistent Planar Memory for Video World Models
Show answer and example
Answer: A few recent frames omit earlier geometry; storing all pixels or points is costly. PPMem maintains compact rasterizable planes across chunks for revisits. Planes approximate scenes and still need occlusion/error management. Persistence alone is not consistency.
Illustrative example: When a camera returns to a desk, persistent memory can preserve a bookshelf that has been absent from recent frames.
Sections in the Chinese source: 背景与主要改动, 核心公式与流程. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why is plane eviction using the next camera path not reading real future video?
Persistent Planar Memory for Video World Models
Show answer and example
Answer: The path can be a provided generation control: it says where the camera will point, not what real pixels will appear. The plane pool prepares for better observations, but the model must generate them. Unknown future observations or gold geometry would be leakage.
Illustrative example: A specified orbit around a table tells where to look without revealing the texture on the back.
Sections in the Chinese source: 背景与主要改动, 核心公式与流程. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
How does a persistent plane pool differ from conditioning only on recent frames?
Persistent Planar Memory for Video World Models
Show answer and example
Answer: Recent context forgets distant views. A world-coordinate pool supplies geometry during revisits and requires occlusion voting, quality deduplication, and eviction. Known future camera pose does not supply real future video.
Illustrative example: A room orbit can reuse an old wall while eviction knows the next camera location, not future pixels.
Sections in the Chinese source: 背景与主要改动, 核心公式与流程. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
Is geometry LoRA enabled at every video denoising step?
Persistent Planar Memory for Video World Models
Show answer and example
Answer: No. Denoise with the frozen stage-one backbone and geometry LoRA off. After generating video, run one extra geometry forward with LoRA to extract planes. Training also uses depth, normals, rasterization, and smoothing, with no rasterization loss in the first thousand steps. Local status is documentation only.
Illustrative example: Generate a 25-frame chunk through multiple steps, then extract planes once and merge the pool.
Sections in the Chinese source: 核心公式与流程, 本地复现与成本判断. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why can uncaptioned video provide a useful language-model midtraining signal?
Mid-Training Language Models on Raw Video
Show answer and example
Answer: Ordered frames contain objects, motion, and spatiotemporal relationships not fully captured by text. The method turns frame patches into continuous sequences and predicts future representations causally. This is not captioning or proof of physical understanding; test after matched tuning.
Illustrative example: Video of lifting a cup preserves motion that the caption “someone picks up a cup” may omit.
Sections in the Chinese source: 背景与主要改动, 核心公式与原文效果. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why does target stop-gradient not freeze the entire vision tokenizer?
Mid-Training Language Models on Raw Video
Show answer and example
Answer: Detaching targets prevents them from moving or collapsing with predictions. The input-side tokenizer still receives gradients through the LM and prediction head. Caption-free midtraining learns visual sequence relations and needs matched instruction tuning for comparison.
Illustrative example: The next patch is detached as target, but prediction error still updates the tokenizer through earlier input patches.
Sections in the Chinese source: 背景与主要改动, 核心公式与原文效果. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
What supervision bottleneck does raw video avoid compared with captions?
Mid-Training Language Models on Raw Video
Show answer and example
Answer: Captions capture only described information, often omitting continuous motion and detail. Continuous token prediction uses raw order but costs longer sequences and vision regression. Better semantics are not guaranteed; match subsequent instruction tuning.
Illustrative example: “A ball moves” omits path and speed variation; raw video contains them, but downstream tests show whether the model uses them.
Sections in the Chinese source: 背景与主要改动, 核心公式与原文效果. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
Why match image-text instruction tuning after video midtraining?
Mid-Training Language Models on Raw Video
Show answer and example
Answer: Final ability depends on both stages. Different later tuning would confound the raw-video effect. The paper matches later tuning, so text-retention conclusions apply to that protocol, not arbitrary captionless training. A local component is not the full comparison.
Illustrative example: Giving only the video arm extra high-quality instructions would confound a later visual QA gain.
Sections in the Chinese source: 核心公式与原文效果, 本地复现与成本决策. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why lighten the entire video-generative robot interface in SmolWAM?
SmolWAM: A Latent World-Action Model for Robotic Manipulation on Edge Devices
Show answer and example
Answer: Vision encoding, video backbone, text conditioning, and action representation all cost compute. A small action head alone can leave most costs. Each change needs compatibility checks and recovery training before real control validation.
Illustrative example: A tiny action head still pays for a large VAE, long visual input, and multistep denoising.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why preserve original token positions and rectified-flow direction when shrinking action queries?
SmolWAM: A Latent World-Action Model for Robotic Manipulation on Edge Devices
Show answer and example
Answer: The pretrained trunk expects particular position semantics and a data-to-noise velocity. A lighter interface that changes them adds distribution shift. SmolWAM restores compatibility stage by stage; checkpoint loading alone does not establish equivalent behavior.
Illustrative example: Reading teacher velocity noise-minus-data with an opposite-sign recovery rule can move actions the wrong way.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
How does a single-frame VAE plus adapter differ from directly replacing the VAE?
SmolWAM: A Latent World-Action Model for Robotic Manipulation on Edge Devices
Show answer and example
Answer: The teacher expects a video-clip latent distribution. The adapter maps single-frame outputs back into that space to preserve a usable trunk interface. Matching shape or token positions alone does not align numeric scale or spatiotemporal semantics.
Illustrative example: Two encoders can output identically shaped tensors but different distributions; the adapter learns compatibility.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
Does SmolWAM use the same rectified-flow velocity direction as LeWAM?
SmolWAM: A Latent World-Action Model for Robotic Manipulation on Edge Devices
Show answer and example
Answer: No. SmolWAM follows teacher data-to-noise interpolation xτ=(1−τ)x0+τε, with velocity ε−x0 and a consistent clean estimate. LeWAM uses the opposite direction. Retain the 12 specified blocks, original positions, and zero-bias sink rather than borrowing another formula.
Illustrative example: Reversing velocity during recovery moves the action in the wrong direction even if loss code runs.
Sections in the Chinese source: 核心公式, 本地复现, 数据协议、结果与边界. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why are action classes different from exact speeds or joint angles?
Time-JEPA: Grounding Video World Models with Physical Time Series
Show answer and example
Answer: Classification can be invariant across speeds and poses, while control and physical analysis need continuous detail. Time-JEPA adds temporal physical supervision without using sensors at deployment. Correct labels cannot establish numeric precision.
Illustrative example: Two clips can both show “raising a hand” with different speeds and elbow angles.
Sections in the Chinese source: 背景与主要改动, 核心公式与隔离要求. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
How can training-only physical time series improve a video-only encoder?
Time-JEPA: Grounding Video World Models with Physical Time Series
Show answer and example
Answer: Auxiliary heads predict speed or joint angles from video and send gradients back to the encoder. Remove the head at deployment; sensor values never enter inputs. Mask invalid frames and fit normalization on training only.
Illustrative example: Train with synchronized speed labels and test on video alone. Supplying speed at test would change the task.
Sections in the Chinese source: 背景与主要改动, 核心公式与隔离要求. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
What does physical supervision add beyond masked latent prediction?
Time-JEPA: Grounding Video World Models with Physical Time Series
Show answer and example
Answer: Latent prediction encourages predictability without ensuring precise physical variables survive. Synchronized labels directly encourage those variables in the encoder. Benefits depend on variable coverage and distribution; one domain's probe does not establish universal physical or control ability.
Illustrative example: A model can predict road texture while ignoring speed differences until a speed target requires them.
Sections in the Chinese source: 背景与主要改动, 核心公式与隔离要求. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
Can a fully missing variable in one tubelet be filled with a valid zero?
Time-JEPA: Grounding Video World Models with Physical Time Series
Show answer and example
Answer: No. Average valid frames independently per variable; empty tubelets remain invalid. Fit and freeze normalization on training. Apply SmoothL1 with β=0.5 only to valid entries, with domain-specific heads, statistics, and weights.
Illustrative example: If neither frame has a joint-angle measurement, skip that variable instead of treating missing as a true zero angle.
Sections in the Chinese source: 核心公式与隔离要求, 本地复现. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why should action token count not directly represent importance across formats?
UniAction: Unified Action Conditioning for World Models
Show answer and example
Answer: Equivalent actions can be tokenized at different granularity. UniAction separates element identity and position and corrects joint attention for count effects. Genuine added actions still carry new information; not every longer encoding is meaningless duplication.
Illustrative example: Splitting a path into 20 rather than ten segments does not double importance, but adding a real grasp matters.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why does adding log(Nref/Na) to action logits cancel identical action-token duplication?
UniAction: Unified Action Conditioning for World Models
Show answer and example
Answer: In joint softmax, copying identical keys, values, and positions increases the action exponent sum. The log-count bias cancels that multiplicity while preserving relative action logits. Different new content or positions can still affect outputs.
Illustrative example: Duplicating each token with identical position would raise uncorrected action weight; corrected output should not change from count alone.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
How does corrected joint softmax differ from separately normalizing video and action then summing?
UniAction: Unified Action Conditioning for World Models
Show answer and example
Answer: Joint normalization shares one denominator and applies log(Nref/Na) to action logits. Separate softmax gives each stream total mass one before merging, a different cross-modal weighting even if internal action ranks match.
Illustrative example: Strong video and weak action evidence can yield low total action weight jointly, but separate softmax forces a full action share.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
Does duplication invariance allow arbitrary added actions or changed positions?
UniAction: Unified Action Conditioning for World Models
Show answer and example
Answer: No. Exact invariance requires identical key, value, and position representations. Na counts valid actions per request; padding is excluded and has zero attention. Supply the native position callback explicitly; local diagnostics are not Wan's native position mechanism.
Illustrative example: Copying the same action at another time changes position, so changed output does not violate invariance.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why can an attractive final image fail the visual reasoning task?
VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning
Show answer and example
Answer: Rules may constrain motion, intermediate states, and forbidden areas. A correct-looking ending can hide process violations. VBVR-Pro turns known rules into executable checks, without measuring undefined quality or assuming every task has a reliable scorer.
Illustrative example: An object can reach its goal by crossing a forbidden area; only the trajectory check catches it.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why prefer an executable scorer to a general VLM judge for some spatiotemporal constraints?
VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning
Show answer and example
Answer: Explicit intermediate states, constraints, and goals can be checked per example rather than by fluency. Executable rewards cover only defined rules. Scorer correctness, gold isolation, and actual trajectory generation still need separate checks.
Illustrative example: Verify each move avoids walls and reaches the target; a fluent explanation does not excuse crossing a wall.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
Is a weighted constraint reward equivalent to requiring all constraints to pass?
VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning
Show answer and example
Answer: No. Weighted sums permit compensation; a hard conjunction makes any violation infeasible. Higher total reward does not guarantee every constraint. This compares objective semantics, not an additional author ablation.
Illustrative example: One of two equally weighted checks gives half credit, while a mandatory safety check cannot be offset by success elsewhere.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
What does the local programmatic reward's gain over a noisy baseline to a perfect score show?
VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning
Show answer and example
Answer: It shows deterministic task scoring and failure attribution on the supplied programs are more stable than a noisy judge analogue. It does not show a multimodal generator gained ability after RLVR. Real trajectories, model updates, and independent evaluation are still needed.
Illustrative example: Scoring a known valid trajectory as one tests the scorer, not a generator's ability to discover it.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why condition robot actions on what the model expects to see?
V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
Show answer and example
Answer: Current images alone may not indicate how the goal changes the scene. V-JEPA Policy connects instruction-conditioned prediction with action learning, giving the action expert predicted context. No future ground truth enters control, and actual execution must still be tested.
Illustrative example: Predict the cup inside a box without reading an actual completion image before the robot moves.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
How does reading predicted future latents differ from reading actual future frames?
V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
Show answer and example
Answer: Predictions come from current observations and instructions and are available at deployment. Real future is training-only supervision. The action expert reads computable foresight, not actual outcomes. Frozen visual encoders do not prevent forecast errors from accumulating.
Illustrative example: Predict a cup's position after grasping; do not feed the real post-grasp photo before judging policy success.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
How does joint world/action training differ from separately training a predictor and action head?
V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
Show answer and example
Answer: Joint training coordinates future context and action flow so predictions can serve control needs. Independent stages need not align. The vision encoder stays frozen; joint predictor/action training does not update every visual parameter.
Illustrative example: Future grasp context can adapt to the action head while the V-JEPA encoder weights remain unchanged.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
How are future-target gradients isolated, and what is locally implemented?
V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
Show answer and example
Answer: Encode actual future with frozen vision and stop gradients. Apply masked L1 and train action velocity. Local work checks future-latent isolation and flow only; no V-JEPA 2.1 checkpoint or robot dataset. Three-seed small tensors do not validate a LIBERO policy.
Illustrative example: No target gradient confirms supervision isolation, not robot success without rollouts.
Sections in the Chinese source: 核心公式, 本地复现, 复现边界. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why isn't WOVEN transition supervision ordinary static classification?
WOVEN: Weaving Visual World Modeling into Multimodal LLMs
Show answer and example
Answer: It models relations among initial state, action, and outcome, including forward prediction, inverse action, and counterfactuals. Generated video rollouts supply visual transitions and intermediate frames, not outcomes stated in prompts. Recognizing object classes is insufficient.
Illustrative example: The same ball starts in one place but moves differently for left and right pushes; choose the action or outcome, not whether there is a ball.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why split WOVEN by initial scene or source rather than random QA rows?
WOVEN: Weaving Visual World Modeling into Multimodal LLMs
Show answer and example
Answer: One image can yield several actions, questions, and counterfactuals. Random row splits can put nearly identical states in both sets. Grouped isolation reduces shortcuts, and scene, action, and reasoning transfer need separate checks.
Illustrative example: Group two questions about different pushes on the same table rather than splitting their common source across train and test.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
What distinct generalization problems do scene-OOD and state perturbation test?
WOVEN: Weaving Visual World Modeling into Multimodal LLMs
Show answer and example
Answer: Scene-OOD changes scene distribution; state perturbation changes state and tests robustness. High in-distribution accuracy does not solve either. The paper also controls action and reasoning axes, and per-benchmark subsets do not establish one universal model gain.
Illustrative example: A new restaurant can be handled well while tiny object-position shifts in one scene still cause errors.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
What do four option permutations and source_id grouping guard against?
WOVEN: Weaving Visual World Modeling into Multimodal LLMs
Show answer and example
Answer: Move the correct letter with its option and image to prevent letter shortcuts. Split by initial source to prevent related variants leaking across sets. The prompt exposes only question, options, and actual images, not answer-related distractor_type. Local training runs SFT, not GRPO.
Illustrative example: Put three questions from one initial frame together in train or holdout so source memorization cannot masquerade as test generalization.
Sections in the Chinese source: 核心公式, 本地复现, 代码映射与边界. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Why doesn't GUI-JEPA generate actions token by token?
GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents
Show answer and example
Answer: It enumerates actions from the accessibility tree and conditions visual latents on each candidate, then scores the batch. The action space comes from observations rather than free-form text generation. Speed depends on candidate count and a11y input, which pixel-only agents may lack, so not all GUI systems are directly comparable.
Illustrative example: Example: Score three valid button actions and choose the best, instead of generating JSON and parsing coordinates.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why route action and ROI through the residual predictor rather than directly to the structured scoring head?
GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents
Show answer and example
Answer: The paper forces candidate-conditioned visual information through the JEPA predictor rather than a shortcut. Action and ROI condition a query; a three-layer Transformer creates a residual latent, followed by ROI and global pooling. Training with a decision loss does not prove accurate future-screen prediction.
Illustrative example: Example: Two click candidates with different ROI content and positions should give different latents. A bypass from raw ROI to the scorer cannot demonstrate predictor utility.
Sections in the Chinese source: 背景与主要改动, 核心公式. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
How does GUI-JEPA scoring differ from predicting a next state and choosing the one closest to a goal?
GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents
Show answer and example
Answer: Latents are decision features trained by listwise action loss; distance to a goal latent is not the value. A randomly initialized predictor can match a dynamics-pretrained one in the paper, so useful decision features are not necessarily good dynamics predictions.
Illustrative example: Example: Confirming may finish a task with little visual change, while an animation looks closer to a target image but is wrong. Action score is not image or latent similarity.
Sections in the Chinese source: 背景与主要改动, 核心公式, 论文离线与线上效果. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
Does a verified three-layer predictor and gradient path mean the repository can operate Android?
GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents
Show answer and example
Answer: No. It receives external screen/condition latents and verifies the residual head, candidate masking and weighted listwise loss. It has no encoder, a11y enumeration, MiniLM ranker or executor. Positive labels are loss-only, not forward inputs. A GPU tensor check is not AndroidControl success.
Illustrative example: Example: Backpropagating on two valid candidate latents only checks gradients. Launching an app also requires screen encoding, candidate discovery, execution and observing the next screen.
Sections in the Chinese source: 本地复现, 检查与负控, 高成本完整实验. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Basics
Does SWIM learn a robot policy or which state coordinates matter for distance?
SWIM: Learning What Matters from Behaviors for JEPA World Models
Show answer and example
Answer: It learns continuous coordinate weights from recorded actions and uses them for world-model geometry. Inverse action prediction only trains the weights. Deployment still needs image encoding, latent dynamics and planning such as CEM. Local coverage is the gate objective and metric, not a robot policy.
Illustrative example: Example: Block position can receive more weight than background light color. That changes distance, not the robot's grasp trajectory.
Sections in the Chinese source: 背景与主要改动, 本地复现. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Core mechanism
Why not gate the current state, and why require true mismatched discrimination?
SWIM: Learning What Matters from Behaviors for JEPA World Models
Show answer and example
Answer: The current state is full action context; the paper gates only future displacement. Fitting positives alone can predict actions from current context and ignore displacement. A cyclic mismatch changes the future while holding current state/action fixed, encouraging behavior-relevant displacement.
Illustrative example: Example: Keep the current position and move-right action. For a negative, take the next batch row's future minus this position. Replacing the current state gives a spurious context cue.
Sections in the Chinese source: 核心公式, 检查与负控. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Method comparison
Are multiplying state by g or sqrt(g), or standardizing again afterward, equivalent?
SWIM: Learning What Matters from Behaviors for JEPA World Models
Show answer and example
Answer: No. Square root scaling yields coefficient g in squared distance; scaling by g gives g². Gates weight already-standardized coordinates; re-standardization cancels scaling. Overall gate scale cancels from Pearson correlation and can be absorbed by the inverse model, so absolute gate magnitudes are not uniquely identifiable.
Illustrative example: Example: Difference 2 with g=0.25 should give 0.25×4=1. Multiplying by sqrt(g)=0.5 before squaring gives 1; multiplying by g gives 0.25. Rescaling the standard deviation afterward cancels the 0.5.
Sections in the Chinese source: 核心公式, 本地复现. The example illustrates the mechanism.
Multimodal models / Visual world models: state, generation, and action conditioning
Implementation details
How do tests distinguish L1 shrinkage from learning behavior, and prevent future state leakage?
SWIM: Learning What Matters from Behaviors for JEPA World Models
Show answer and example
Answer: Check gate gradients from the behavioral term alone, then real optimizer updates; parameter change alone is insufficient. In stage two state distance is stop-gradient. Future state constructs offline displacements only and must not be available to deployed planning. Constant-distance Pearson is undefined and should be rejected, not declared valid via arbitrary epsilon.
Illustrative example: Example: λsum(g) shrinks gates even with actions ignored, so also test λ=0. If all frame distances are zero, reject the zero-denominator correlation rather than report success 1.
Sections in the Chinese source: 本地复现, 检查与负控. The example illustrates the mechanism.
Topic guide and reading order
Start with this question¶
Does the model only recognize a scene, or can it predict how an action changes the state?
Takeaway¶
Learning prediction or control representations requires observations, actions, and future states together. LeWAM jointly learns latent future prediction and action flow matching so representation and policy shape each other. Planning uses latent dynamics to evaluate actions. WOVEN organizes generated video transitions as supervision controlled by scene, action, and reasoning type for consequences, causes, and counterfactuals. SWIM asks which state coordinates should contribute to latent distance. Offline inverse dynamics gates only future displacement; once frozen, sqrt(g) calibrates distances between same-frame pairs. Planning does not read future states.
Example¶
Pushing a cup left and right on the same table should lead to different outcomes. Recognizing a cup does not control a robot. Latent prediction can avoid pixel reconstruction, but test whether actions matter, whether multistep rollouts drift, and whether selected actions reach goals in closed loop.
Common pitfall¶
Realistic videos, low one-step error, or high training-set QA scores do not prove effective planning. Separate source scenes and trajectories by group and prevent counterfactuals from leaking future results. Local LeWAM is a non-held-out closed-loop diagnostic; WOVEN is a small experiment on public training data. Neither establishes the paper's full generalization or robotics claims.
Reading order¶
Start with LeWAM's joint latent and action objective to see how prediction feeds control. Then read WOVEN on visual transitions and controlled supervision. Distinguish representation learning, synthetic data, question answering, and closed-loop planning. Follow up with visual geometry, video memory, or embodied actions.
- LeWAM: End-to-End World and Action Modeling with JEPAs · Original paper
- WOVEN: Weaving Visual World Modeling into Multimodal LLMs · Original paper
Check your understanding¶
Does lower one-step latent MSE necessarily improve long-horizon control success?
Show suggested answer
No. Errors can fall in directions irrelevant to action choice, compound over a rollout, or be exploited by policy search. Compare closed-loop control with matched environments, goals, and budgets, and do not present training trajectories as held out.