World Models vs Video Prediction: What I Got Wrong

When Genie 4's demos landed, my first reaction was the same one I've had since Sora: this is a very good video generator, why are we calling it a world model? A day of poking at the clips and reading the reactions of people smarter than me changed my mind about the framing — and reminded me of three mistakes I've made building and evaluating world models in simulation work. This isn't a review of DeepMind's release. It's a set of lessons learned the expensive way about the difference between predicting video and simulating an environment, because the demos make it easy to conflate them.

Lesson 1: A World Model Is a Transition Function, Not a Movie

The mistake I made for a long time — and the one I see in most hot takes on Genie 4 — is evaluating interactive simulation with the rubric we use for video generation. Prompt goes in, clip comes out, we judge fidelity, motion smoothness, prompt adherence. Done. That rubric misses the entire point of the thing being built.

A video generator learns p(frames | prompt). A world model learns something closer to p(next_state | current_state, action). The distinction sounds academic until you stress it. In video prediction, nothing you do during playback changes anything; there is no 'you'. In a world model, an agent (or a user with a keyboard) injects actions at every step, and the model must condition on them. The moment you allow interaction, you've moved from a generative art problem to a control-theory-flavoured problem: the system must respond to inputs it was never shown, in states it drifts into on its own, indefinitely.

This is why Genie 4's keyboard navigation and mid-scene text steering matter more than its photorealism. Photorealism is table stakes in this generation of models; Sora 2 and Veo 3 already cleared that bar. Real-time action conditioning — press a key, the world responds causally and keeps responding coherently — is the actual research contribution. When I see people score the demos purely on image quality, they're grading a flight simulator on how nice the windows look.

The practical fix for the confusion: when evaluating any claimed world model, ask three questions before looking at a single pixel. Does it take actions at inference time, or only a prompt? Does the same action in the same state do the same thing? And what happens if you poke it ten thousand steps in a row? Those three questions separate environment simulators from very impressive televisions.

Lesson 2: Off-Screen Consistency Is Where the Lie Lives

The sharpest observations in the public reaction weren't about image quality at all. People noticed that in the dragon-racing demo, when the camera swings back toward the castle, a chunk of its right side — catapult included — has vanished, and a second castle seems to have appeared. Pedestrians walking away from the camera on the train platform change identity when the camera returns. A moped rider gains a helmet between cuts of attention. The model maintains consistency beautifully for whatever stays in frame, and quietly improvises whatever doesn't.

I've built systems with exactly this failure mode, so I recognise it immediately. It's what you get when the model's state is essentially a sliding window of recent latent frames plus whatever the attention mechanism can carry. There is no explicit map, no object permanence store, no scene graph. The model doesn't know there's a castle; it knows the last several thousand tokens looked castle-ish, and once those tokens scroll out of context, the castle's existence becomes a coin flip weighted by priors. Castles are common in dragon scenes, so a new one materialises.

A model that forgets what left the frame hasn't learned a world. It's learned a very confident, very plausible present.

Here's the uncomfortable part: this might be fine for entertainment and completely disqualifying for agent training. If you're training a robot policy inside a simulator, the policy will learn whatever the simulator rewards. In a simulator without object permanence, an agent can 'solve' tasks by waiting for the world to reroll the dice — walk away from an obstacle and it may not exist when you turn around. The resulting policy transfers to reality about as well as a policy trained in a video game with broken collision detection. I've watched teams burn months on sim-to-real gaps that traced back to exactly this class of inconsistency.

The fix, conceptually, is memory as a first-class architectural component rather than an emergent property of context length. Whether that's an explicit spatial representation (NeRF-style scene state, voxel maps, object slots) or a much longer and smarter latent memory, the model needs to be able to answer: what is behind me right now? without re-dreaming it. DeepMind leaning on Gemini's reasoning models to 'steer and monitor' scene coherence reads to me as an admission of this — coherence currently has to be babysat by another model because the generative core doesn't hold it natively. That's a pragmatic patch, not a solution, and I mean that as a description, not an insult. Pragmatic patches are how real systems ship.

Lesson 3: Physics in a Latent Space Is a Prior, Not a Solver

The release materials emphasise aerodynamics, collisions and natural human motion, and the watermelon-smashing and falling-seed demos do look physically fluent. My second big mistake, historically, was assuming that fluent-looking physics in a generative model means the model has internalised physics in a way you can rely on. It hasn't. It has learned the visual statistics of physics from enormous quantities of video, which is a very different thing from a constraint system.

The tell is distribution shift. Latent physics models nail the hundredth-percentile-common cases — a ball bounces, water ripples, fabric billows in wind, because the training data contains millions of these. Push toward the tail and the priors crack. One demo prompt fought the model hard to avoid a 'video game look' despite begging for photorealism, because a peacock transforming into a cat via magic wand is so far off-distribution that the model slid toward the nearest plausible genre in its priors, which apparently is stylised 3D animation. Physics learned as texture fails exactly where you'd most want a simulator: the novel situations you can't film.

The skateboarder test — make a line that fools an actual skater — is the right instinct for the same reason. Skaters carry an embodied physics engine tuned by thousands of falls. Weight transfer, pop timing, the way momentum trades against friction on concrete: get any of it subtly wrong and it reads as wrong to an expert even when it looks fine to everyone else. Current latent world models don't pass domain-expert scrutiny on mechanics, and I'd bet against any purely video-trained model passing it soon. The physics is in the data's shadows, not in the weights as an enforceable law.

My rule of thumb now: treat a latent world model's physics like a talented storyboard artist, not an engineer. It will draw you a convincing explosion. It will not guarantee the shrapnel conserves momentum. If your use case needs guarantees — contact-rich robot manipulation, fluid-structure interaction, safety validation — you still need a real physics engine, and the interesting work is in hybrid systems. This is the same pattern I argued for with hybrid reasoning agents: let the learned model handle the high-dimensional fuzzy stuff, let a verifiable component handle invariants.

Lesson 4: Don't Train Agents Where the Ground Truth Is a Mood

Every world-model announcement includes the robotics pitch, and Genie 4 is no exception: infinitely variable training environments for robots and agents. Every time, a chunk of practitioners responds with justified skepticism — you can't train a robot in a dream. Both sides are half right, and the resolution is specificity about which agents and which skills.

For low-level motor control, latent simulators are currently a trap. A manipulation policy needs force feedback, contact dynamics, and repeatable physics; a model that rerolls off-screen geometry and approximates collisions from video statistics teaches brittle, exploitative policies. Anyone shipping sim-to-real for grasping should stay on MuJoCo, Isaac or Bullet-class simulators and treat the world-model demos as a research direction to watch.

For high-level decision-making, though, the calculus flips. Navigation policies, driving scenario testing, web-and-UI agents, game-playing agents, multi-step planners — these mostly need plausible consequences of actions, not Newton-exact dynamics. We've already seen the pattern work: model-based RL systems like Dreamer train inside a learned latent dynamics model and transfer to real Atari and control tasks; Waymo and others have long used learned behavioural models of other road users inside their sims. A generative world model is that idea scaled to open-ended visual environments. The mistake is demanding one model serve both the planner and the servo loop.

• Use latent world models for: scenario diversity, rare-event generation, high-level policy evaluation, human-in-the-loop testing, synthetic data for perception.
• Use physics engines for: contact-rich manipulation, dynamics-sensitive control, anything where an inconsistent collision once in ten thousand steps poisons the reward signal.
• Use neither blindly: validate the policy against reality early and often, because every simulator — learned or hand-built — has a gap, and you want to find yours in week two, not month nine.

Lesson 5: Real-Time Audio and Dialogue Change the Product, Not the Science

Genie 4 is the first of these models to generate ambient audio and interactive dialogue natively, and the internet immediately caught it playing a clarinet when asked for a saxophone. Funny, and also instructive. Cross-modal consistency is the same object-permanence problem wearing a different hat: the audio head has its own latent notion of the scene, and 'brass single-reed instrument' is close enough in embedding space that the mix-up slips through. Expect a long tail of these — footsteps on snow that sound like gravel, dialogue that answers a question the scene has already moved past.

Strategically, though, audio is a bigger deal than the saxophone gaffe suggests. A silent world is a demo; a sounding world is a product. The natural-language steering — change the weather, add a person, mid-stream — is the other half. Together they turn the world model from a research artefact into an interactive medium, which is clearly where Google wants this: Project Genie, AI Ultra subscriptions, API partners. The 'movies become video games' prediction is the obvious commercial endgame, and for once the hype rhyme scans.

For practitioners, the lesson is to separate the two threads in your head. The product thread (real-time, steerable, audiovisual, monetisable) is moving fast and will produce genuinely useful tools for prototyping, previsualisation and interactive storytelling within a year or two. The science thread (persistent state, causal physics, reliable long-horizon consistency) is moving at research pace and should not be assumed solved because the demo is gorgeous. I've made the mistake of reading a product milestone as a science milestone before; it's how you end up architecting a system around a capability that doesn't exist yet.

What I'd Actually Do With This Today

If you're building agents, the pragmatic play hasn't changed: keep your control stack on real physics, but start experimenting with learned simulators for the layers above it — scenario generation, policy smoke-testing, and synthetic perception data. The cost curve here is falling the same way it did for LLM inference, and as I wrote in why AI benchmarks miss real work, the teams that win are the ones evaluating on their actual task distribution rather than public demos. Build a harness that rolls the world model out for thousands of interactive steps and measures consistency failures — object reappearance, geometry drift, causal violations. That harness will tell you more than any launch video.

Also keep an eye on the open-weights side — Tencent's HY-World 2.0 got a fraction of Genie 4's attention, and if history with LLMs and image models is any guide, the open ecosystem is where practitioners will actually learn how these systems tick. You can't introspect a research preview behind a waitlist. You can fine-tune, probe and break something you can download, and the probing matters: as with mechanistic interpretability work, the interesting findings come from poking the network, not admiring it.

The honest summary: Genie 4 is a real step on the hard axis — interactivity — not just the easy axis of prettier pixels. But the gap between 'maintains a scene while you look at it' and 'simulates a world' is still wide, and it's exactly the gap that matters for agent training. Call it a world model if you like. Just remember that a world, unlike a video, is still there when you turn around. These ones, mostly, aren't — yet.