Predicting the next scene is not the same as acting or preserving a world¶

Updated 2026-10-30 · 24 topic entries / 8 representative comparisons

World models must separately show what state retains, how actions alter futures, how plans are evaluated, and how memory persists across segments. LeWAM and V-JEPA Policy connect prediction and actions; prediction–planning evaluation decomposes errors; WOVEN supervises visual transitions; PPMem maintains geometry; VBVR-Pro executes task checks. SWIM learns relative state-coordinate weights from offline action predictability to shape latent distances without giving deployment planners real future states.

Mechanisms · Trade-offs · Cross-paper questions · Updates

How to read this card

The table compares representative methods. It is not a shared-benchmark leaderboard. Teaching examples are not measured results.

Mechanism comparison¶

Axes: Future latents, actions, visual transitions, or persistent geometry · Jointly shaped or frozen vision encoders · Latent loss, physical state error, or actual planning success · Candidate generation, executable constraints, and cross-view consistency

Paper and source	Mechanism	Benefits and trade-offs
LeWAM: End-to-End World and Action Modeling with JEPAs · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 数据协议、结果与边界	Jointly train the visual encoder, action-conditioned latent dynamics, and flow-matching policy, with refinement of candidate plans.	Control can shape representations but predictions and planning still need separate tests. The scaled A100 closed loop was non-holdout and found no planning advantage.
V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Freeze V-JEPA 2.1 vision; jointly train a future latent predictor and action expert conditioned on predicted context.	Uses pretrained vision without control gradients changing it. Local work only checks L1 stop-gradient and flow components, not the original model or robot data.
One-Step Prediction Fidelity Is a Poor Proxy for Planning Performance in JEPA World Models · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 预算与实现差异, 聚合、结果与边界	Use one frozen physical decoder on real and predicted latents to separate representation, prediction, and excess error, then evaluate planning independently.	One-step errors do not measure long-horizon success. Full multi-environment retraining/MPC was too expensive; CPU results only diagnose interfaces.
WOVEN: Weaving Visual World Modeling into Multimodal LLMs · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 数据协议、公开资源与结果	Organize actually generated video rollouts as `(s,a,s′)` for forward, inverse, and counterfactual visual transition supervision of an MLLM.	Transition fidelity and OOD tasks need independent checks. A small local real-transition SFT ran, but later GRPO did not.
Persistent Planar Memory for Video World Models · Original source Sections in the Chinese source: 背景与主要改动, 核心公式与流程, 本地复现与成本判断	Merge local planar primitives into a persistent world-coordinate pool and feed rasterized depth into later video chunks.	Geometry compression must address occlusion, deduplication, and cameras. Research documentation only; full video/geometry training was too costly.
VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 复现边界	Use executable task-specific scorers for intermediate and final state and constraint checks, supplying reward to generators.	Deterministic rules cover defined tasks only. Local scorers are not trained 14B/19B video models.
GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents · Original source Sections in the Chinese source: 背景与主要改动, 核心公式	Predict candidate-action-conditioned residual successors and score ROI/global structure with a listwise positive loss.	Real screen encoding, candidate generation, and environment are external. No GUI benchmark capability result.
SWIM: Learning What Matters from Behaviors for JEPA World Models · Original source Sections in the Chinese source: 背景与主要改动, 核心公式, 高成本完整实验	Predict fixed recorded action chunks from current state and gated future displacement. Learn gates with NLL, cyclic discrimination, and L1, then calibrate latent distances with `sqrt(g)` weights.	Offline behavior labels are not planning inputs. Gates encode relative weights and must not be standardized again. No local visual encoder, SIGReg, environment, CEM, or full training.

Trade-off analysis¶

Readable state, predictable futures, and successful action are separate checks¶

LeWAM jointly shapes vision with control and prediction; V-JEPA Policy learns prediction/actions on frozen vision. Either way, test if latents retain physical state, how much additional error prediction adds, and whether planning succeeds. A common physical decoder separates the first two; planning is run independently rather than inferred from latent loss.

Sources: LeWAM: End-to-End World and Action Modeling with JEPAs · V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents · One-Step Prediction Fidelity Is a Poor Proxy for Planning Performance in JEPA World Models

Generating plausible transitions and verifying consistency need separate interfaces¶

WOVEN supplies generated video transitions whose visible changes must actually supervise learning instead of revealing outcomes in prompts. PPMem preserves geometry across chunks and views; future camera paths do not reveal real future frames. VBVR-Pro checks executable constraints, but a scorer does not train a capable generator. These are data, memory, and verification respectively.

Sources: WOVEN: Weaving Visual World Modeling into Multimodal LLMs · Persistent Planar Memory for Video World Models · VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning

October 29: new method and evidence boundary¶

GUI-JEPA predicts residual successors in current screen latents and combines action-specific ROI/global regions and structural cosine scores. This differs from pixel-video supervision and actual GUI execution. The component receives external frozen latents and text matching scores, retains a trained three-layer predictor and head, and does not use positive training labels to choose actions in forward inference.

Sources: LeWAM: End-to-End World and Action Modeling with JEPAs · GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents

Behavior can shape geometry without becoming a future-state oracle¶

GUI-JEPA predicts candidate-conditioned latent residuals and learns listwise decision scoring. SWIM uses offline action chunks to learn which state displacements predict those actions, then freezes continuous gates as supervision for latent geometry. Neither latent check establishes execution. SWIM's h changes future-state spacing, not the fixed current action chunk. Inverse dynamics and true future states stay out of planning; coordinate standardization after sqrt(g) cancels the weights.

Sources: SWIM: Learning What Matters from Behaviors for JEPA World Models · GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents

Cross-paper questions¶

Try answering before expanding the explanation. Each question relates at least two papers.

Do LeWAM and V-JEPA Policy differ only in their visual backbone?¶

Show answer and example

Answer: No. LeWAM jointly updates encoder, dynamics, and policy with online targets; V-JEPA Policy freezes pretrained vision and detaches future targets. Whether control gradients shape vision and whether targets move are different training contracts.

Illustrative example: Grasping supervision can adjust vision in the joint path, while the frozen path mainly changes prediction and action modules.

Sources: LeWAM: End-to-End World and Action Modeling with JEPAs · V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents

Does lower JEPA prediction error show LeWAM candidate planning beats a reactive policy?¶

Show answer and example

Answer: No. Prediction error mixes latent readability and added dynamics error; plans also depend on action candidates, accumulation, and goals. Decode with the same physical head to separate errors and test planning independently. Local non-holdout LeWAM did not observe an advantage.

Illustrative example: Predicting a static scene accurately does not choose actions that reach a target.

Sources: One-Step Prediction Fidelity Is a Poor Proxy for Planning Performance in JEPA World Models · LeWAM: End-to-End World and Action Modeling with JEPAs

Which of WOVEN's videos, PPMem's geometry, and VBVR-Pro's scorer guarantees correctness?¶

Show answer and example

Answer: They serve different roles: action-conditioned transition data, persistent memory, and executable checks. Realistic video does not guarantee a valid transition, stored geometry can err, and scores cover only defined rules. Verify data, state maintenance, and scoring independently.

Illustrative example: Memory may preserve object position at an old viewpoint, but counting and ordering require their own validation.

Sources: WOVEN: Weaving Visual World Modeling into Multimodal LLMs · Persistent Planar Memory for Video World Models · VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning

Why isn't a GUI-JEPA successor prediction a successful action verified on a phone?¶

Show answer and example

Answer: A learned latent can omit real state constraints. Success needs environment feedback; small latent forward passes, candidate permutation equivariance, and gradients verify only the module.

Illustrative example: A predicted page vector near the target does not show that a real button was clickable or the request succeeded.

Sources: LeWAM: End-to-End World and Action Modeling with JEPAs · GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents

Why may neither SWIM nor GUI-JEPA give a real successor state to a deployment policy?¶

Show answer and example

Answer: q(t+h) is offline supervision for SWIM inverse dynamics and gating; only weighted latent geometry transfers to image-based planning. GUI-JEPA likewise predicts rather than observes candidate successors. Real successors would turn the evaluation into an oracle.

Illustrative example: Offline future frames can label displacements, but action selection cannot see those answers. Varying h also does not stretch the fixed five-frame action target across the entire h interval.

Sources: SWIM: Learning What Matters from Behaviors for JEPA World Models · GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents

Update log¶

Updates record new approaches, changed trade-offs, and follow-up evidence.

2026-10-30 · Reviewed corrected SWIM: distinguish fixed action chunks from h, use ungated current state, cyclic negatives, real gate gradients, and sqrt(g) geometry transfer. Compare GUI-JEPA candidate residuals without exposing future training states to planners.

Affected: SWIM: Learning What Matters from Behaviors for JEPA World Models · GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents

2026-10-29 · GUI-JEPA predicts action-conditioned residual screen successors and uses ROI/global structural cosine scoring, not pixel generation or a real GUI executor. The component takes external frozen latents and text scores, implements the three-layer predictor and head, and does not use positive training labels to make forward choices.

Affected: GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents

2026-10-27 · Initial version: compared joint and frozen vision for action, prediction versus planning tests, WOVEN transitions, PPMem geometry, and VBVR program scores; screened 22 abstracts and read six papers.

Covers 22 topic entries; see the full list below.

All 24 topic entries

The full catalog follows. The main comparison covers the representative papers listed above.

Paper-by-paper questions · All deep dives