Physics-Consistent Video Generation

Topic overview

Physics consistency means that motion, interaction, and state change in a generated video follow real-world physical laws. Temporal smoothness, appearance continuity, natural motion, and high visual quality are useful prerequisites, but they do not by themselves establish correct gravity, contact, material response, conservation, or causal order.

Why now?

Recent work has moved beyond the simple recipe of letting a simulator compute motion and a generator render it. Physical information may come from simulation, real video, explicit properties, foundation-model features, VLM feedback, measurable laws, or the generator's own latent states. It may enter during planning, training, conditioning, sampling, or evaluation. Benchmarks have expanded at the same time, so “better physics” can no longer be summarized by one score.

Background: three distinctions

Source × stage: simulators, data, VLMs, learned representations, and physical laws can enter before generation, during training, as explicit conditions, or at inference time.
Model × system: updating the video backbone, training an auxiliary dynamics or reward model, and adding test-time search are distinct interventions.
Motion outcome × physical condition: masks, flow, and trajectories often specify how an object moves; force, mass, friction, restitution, and material properties ask the system to infer the resulting motion.

Method landscape

Explicit dynamics and control: PhysGen and related systems infer state, simulate motion, then use a video model for rendering or refinement. Their constraints are interpretable, but hidden geometry, material, contact, and force are often underdetermined.
Data, representation, and reward: PISA teaches a controlled behavior through post-training; VideoREPA borrows spatiotemporal relations from a video encoder; VLM or law-based rewards optimize generated motion. These methods change the generator, but may still learn templates or evaluator shortcuts.
Latent dynamics and inference-time alignment: some methods add physical state to the prediction target, while others use a frozen world model, intermediate features, or early motion priors to search, select, or refine samples without updating the generator.
Evaluation: open-ended judges offer breadth, real futures provide controlled evidence, equations test measurable laws, and likelihood preference probes internal model ranking. These protocols answer different questions.

Representative evidence and its boundary

PhysGen shows that explicit state estimation and simulation can guide an image-to-video pipeline, but the guarantee belongs to the full pipeline and depends on the inferred state. PISA shows that small amounts of simulated data and reward can change free-fall behavior while exposing weak out-of-distribution generalization. VideoREPA improves physics benchmarks through representation alignment, yet assumes that the teacher features encode transferable physics rather than stronger visual statistics. Verifiable rewards are easier to audit but cover fewer phenomena; inference-time alignment can find better samples without proving that the generator learned new knowledge.

Failure cases

Current methods degrade with hidden initial state, contact-rich interaction, material change, deformation, long horizons, unseen parameters, and combinations of physical rules. Explicit pipelines inherit state-estimation errors; control may be confused with physical inference; reward optimization may exploit an evaluator; and inference-time search only works when the candidate distribution already contains a better trajectory.

Connections to our research

The topic connects long-video generation, causal and autoregressive modeling, controllable generation, world models, and video evaluation. Long rollouts amplify small dynamics errors, while shared reward and evaluation models can create circular evidence.

Research-quality prompts

Central claim: Progress comes from better interfaces that turn physics into controllable and testable supervision, not from one route replacing all others.

Hidden assumption: Trajectories, flow, VLM scores, learned features, and video likelihood can separate physical mechanisms from visual correlation.

Failure boundary: Evidence remains strongest for short, observable, single-event, and controlled phenomena.

Connection: Every claim should identify what changed in the generator, auxiliary modules, and inference system.

Concrete experiment: Fix the backbone, data, and compute, vary one supervision interface, then test unseen parameters, appearances, rule combinations, and longer horizons.

Discussion seed: If inference-time search reliably finds better physics, is that model capability, latent capability in the distribution, or system capability?

Open questions

Which parts of geometry, material, contact, and force are identifiable from one image?
How can evaluation separate following a supplied trajectory from inferring motion from physical conditions?
Should physical state be explicit and writable, or remain inside a generator's latent representation?
What experiment would show that latent dynamics are stable and intervenable rather than merely benchmark-useful?
How can training rewards and final evaluators remain independent enough to avoid circular validation?
How should a benchmark represent multiple physically valid futures and long-horizon error accumulation?

Possible project

Build a small counterfactual benchmark that fixes scene and appearance while changing one physical variable at a time. Compare simulated-data fine-tuning, representation alignment, preference optimization, verifiable reward, and inference-time search on the same backbone, and report both law-level error and system cost.