PEIYUAN ZHU · VIDEO ACTION EDITING

Action Editing
VADD

VADD · Data generation, identifiable action anchors, and qualitative evidence

01

Change the requested action while preserving source content that the edit does not target.

THEORY invariance and action informationMETHOD source-conditioned action adaptersINFERENCE self-generated demonstrations

VADD learns action anchors from demonstrations and trains an editor to execute them under source conditioning. The data-generating process, identification result, method, and video evidence follow.

IMotivation

IIIdentifiable action anchors

IIIVADD editor

IVQualitative results

I · MOTIVATION

Change What Happens—Preserve Who and What

An action edit is valid only when the requested event occurs with the source-owned subjects and objects.

02

FM-1 · NO EDIT

The requested event never starts

The model is unresponsive: the source action continues, or the edit arrives only after the incompatible source event has already been replayed.

FM-2 · INVENTORY / INTERACTION

The object appears on demand

A manipulated object materializes, disappears, or bypasses the intended physical relation instead of remaining the same source-owned instance.

FM-3 · SUBJECT / ROLE REPLACEMENT

The interaction uses the wrong identity

Motion occurs, but the original actor or patient is replaced, re-rendered, or rebound. Plausible interaction cannot compensate for subject-preservation failure.

Non-compensatory gates

Action response, same-instance continuity, interaction topology, and subject preservation must all pass.

I · MOTIVATION · SELF-GENERATED DEMONSTRATIONS

Recover Action Capability from Demonstrations

Self-generated anchors reveal missing action mechanics; the editor should add only their source-faithful counterparts.

03

Original VADD manuscript diagram: self-generated demonstrations guide recovery of missing action capability under source preservation. — **Conceptual support-space view.** Shapes encode capability-set relations only; their area, distance, and anchor count do not represent probability mass.

Provisional design

01 · BASE EVIDENCE

Generation competence ≠ editing competence

Orange anchors show that frozen T2V can generate an action mode that source-conditioned editing may still miss.

02 · TRANSFER RULE

The anchor is evidence—not the target

Its actor, objects, scene, and camera may differ. Only qualified event mechanics can transfer.

03 · LOCAL ADDITION

Recover the source-faithful counterpart

The recovered action must express the requested behavior while preserving source-owned identity, inventory, background, and unrelated motion.

I · MOTIVATION · DIRECT FAILURE EVIDENCE

Failure Mode 1 — The Requested Event Never Starts

Source-conditioned output can preserve source motion even when an action-only anchor makes the requested edit explicit.

04

Established evidence

Jellyfish drift → bell contraction

REQUESTED EVENT NEVER STARTS

Jellyfish drift → bell contraction

Real target contracts into a ring and the anchor demonstrates strong bell-shape change. Frozen keeps the source jellyfish rounded and drifting; contraction never begins.

**Frozen base · rounded drift persists**

Ordinary phone use → playful selfie

REQUESTED EVENT NEVER STARTS

Playful selfie

Target and anchor lift the phone and add tongue/wink behavior; frozen continues ordinary phone use.

Direct evidence for FM-1

Bell contraction and the playful selfie exist in the target and action-anchor branches, but neither requested transition starts reliably in frozen source-conditioned editing.

I · MOTIVATION · FULL-SEQUENCE EVIDENCE

A Late Target-Like Endpoint Is Not an Edit

Frozen may reach part of the requested endpoint only after replaying most of the incompatible source sequence.

05

Established evidence

Dance in place → full scarf spin

DELAYED EDIT

The spin arrives after the dance

Frozen replays source-like in-place scarf motion for most of the clip, then begins a late turn. A late spin does not replace the earlier source event.

One-arm face cover → eye rub and two-hand cover

DELAYED EDIT

The second hand arrives at the end

Frozen holds the source one-arm cover through most of the sequence, then adds the second hand late. Ordered eye-rubbing motion remains absent.

Onset and order are non-compensatory

Sparse endpoint frames can look compliant while the full video remains source-driven. Later target similarity cannot erase an incompatible event that already occurred.

I · MOTIVATION · MANUAL FAILURE EVIDENCE

Failure Mode 1b — Source Motion Is Replayed Before the Edit

A delayed or partial response is still failure when the incompatible source event has already occurred.

06

Established evidence

Turn and place the bottle on the treadmill

SOURCE MOTION IS REPLAYED

Bottle placement

Target and anchor turn and place the bottle; frozen keeps holding it and facing forward.

Bow → stand upright and clap

SOURCE MOTION IS REPLAYED

Bow → clap

Frozen first inherits the complete bow, then only partially brings the hands together near the end.

Source-motion leakage

These frozen-editor examples retain the original source action. Check action execution together with subject and scene preservation.

I · MOTIVATION · PAPER AUDIT

Self-Generated Demonstrations vs. Edited Outputs

128 source–target pairs · frozen Bernini-R 1.3B

07

Ordered V-JEPA2 action margin and pairwise preference
Candidate	Mean $m_act$	Metric preferred	Human preferred
Self-generated T2V	−0.01031	125 / 128	114 / 128
Source-conditioned edit	−0.14660	3 / 128	12 / 128
Tie	—	0 / 128	2 / 128

PAPER UPDATE · 01 / 06 · DATA GENERATING PROCESS

Data Generating Process: Content × Action

VADD: Video Action Editing via Demonstrations and Disentanglement

08

Controlled dynamical process and a two-by-two content-action recombination diagram

GENERATE

Roll out an ordered trajectory

Content $z^c$ is independent of the initial state, action regime $u$ , and innovations. The renderer combines content with the full realized trajectory $z^a_{0:T}$ .

EDIT

Recombine content and action

Preserve source content; transfer the demonstration’s trajectory, including its initial state and timing, within a fixed physical context.

SIMULATE

MuJoCo supplies controlled views

Replay physical motion under two block appearances. Physics evolves the objects; a fixed camera and lighting render each clip.

§2.1 + Appendix D · The strict controlled model fixes physical context; real consecutive-event clips approximate this coupling. Manuscript source ↗

PAPER UPDATE · 02 / 06 · IDENTIFIABILITY

When Does an Action Anchor Identify Action?

The positive pair shares the entire realized trajectory, not only the action label.

09

ACTION-PRESERVING VIEWS $V = g_ctrl(z^c, F_T(w)), V⁺ = g_ctrl(z^{c+}, F_T(w)). Both views share the full realized trajectory.$

POPULATION OBJECTIVE

J(E_a, P) = λ E‖E_a(V) − E_a(V⁺)‖² + E‖P(E_a(V)) − φ(V⁺)‖²_H_k, λ > 0. The two terms are labeled L_inv and L_pred.

Agreement removes appearance variation. Prediction of a characteristic kernel embedding retains differences between action-conditional video distributions.

Conditions for the theorem

Regularity: compact supports and continuous dynamics / rendering.
Appearance variation: independent, full-support content resampling, independent of the shared trajectory.
Temporal observability: distinct trajectories cannot produce the same ordered video under supported content choices.

Optimize over continuous encoders, finite-risk predictors, and a code dimension admitting continuous injective trajectory coordinates.

AT EVERY POPULATION GLOBAL OPTIMUM

E_a(V) = h_a(z^a), P(h_a(z^a)) = μ_k(z^a), almost surely.

$h_a$ is injective on a full-measure action subset; $μ_k$ is the action-conditional kernel mean embedding. Individual semantic coordinates are not fixed.

§2.2 · Original manuscript equations. LaTeX source ↗ · Finite-data optimization and real-video patch views are outside this guarantee. Manuscript source ↗

PAPER UPDATE · 03 / 06 · TEMPORAL INFORMATION

Why the Ordered Video Matters

Three observations identify position, velocity, and acceleration; two can remain ambiguous.

10

NONLINEAR OBSERVATIONS

p_t = p_0 + t v_0 + ½ t² u, x_t = ρ(c, p_t).

Temporal observability concerns the whole ordered clip. A single frame need not reveal the full action state.

Why ordinary mean prediction can fail

The multiplicative map $ρ(c,p) = (cp,c)$ with symmetric nonzero content is invertible, yet $E[V | z^a] = 0$ . Predicting this mean can erase all action differences.

Manuscript numerical check · $T = 3$
Mixing / objective	$p$ $R²$	$v$ $R²$	$u$ $R²$	Content $R²$
Shear · agreement + kernel	1.00	1.00	0.97	0.00
Multiplicative · same objective	1.00	1.00	1.00	0.00

When feature means suffice

In the additive-feature corollary, action differences must survive outside the appearance-variation subspace. Ordinary feature prediction then retains action.

§2.3 + Appendix E · $R²$ values are three-seed means from the paper’s numerical experiment, separate from rendered-video examples. Manuscript source ↗

PAPER UPDATE · 04 / 06 · REPRESENTATION LEARNING

Learning Action Anchors from Demonstrations

A temporal model keeps ordered dynamics across spatial views of the same demonstration.

11

Paper VADD architecture: action anchor training, condition K, source-conditioned adapters, and behavior supervision

01

Observe dynamic features

Frozen features → temporal centering, signed changes, and local change magnitudes. Alternating patches supply two views.

02

Encode an action sequence

A bidirectional GRU predicts Gaussian posteriors. Their means are the action anchors; a recurrent, history-conditioned prior regularizes them.

03

Retain information and order

SmoothL1 dynamic reconstruction + view consistency + KL / temporal regularization and reverse, shuffle, repeated-frame controls.

ACTION-ANCHOR OBJECTIVE

L_anchor = λ_obs L_obs + λ_view L_view + L_reg.

§3.1 · Same-scene patch views and mean-target SmoothL1 are practical surrogates for the population criterion. The prior is a temporal inductive bias. Manuscript source ↗

PAPER UPDATE · 05 / 06 · EDITOR

Source-Conditioned Action Injection

Action specifies the motion; the source provides the appearance and spatial context in which it is executed.

12

DEMONSTRATION $V_r$ Normalized anchor

learned ordered action

+

DEMONSTRATION $V_r$ Compressed V-JEPA2

complementary behavior features

→

DETACHED CONDITION $K(V_r)$

phase-aligned feature concatenation

SOURCE-CONDITIONED LOW-RANK UPDATE

Δu = α_k W_up [(W_u u) ⊙ tanh(W_k k) + (W_k k) ⊙ (1 + tanh(W_b b))].

GENERATED STATE · $u$

Adapt to the current token

The first term couples the demonstrated action to the current generated state.

SOURCE CONTEXT · $b$

Bind the action to the source

The second term modulates the action using spatial context from independently encoded source frames.

SAMPLING GATE · $α_k$

Inject early, refine later

Adapters act on visual self-attention value / output projections during the first 8 of 40 sampling steps.

Scope of the theory

The identified object is the learned mean code. Normalization can discard offsets and scale; the augmented packet $K$ also includes V-JEPA2 features.

§3.2 + Appendix D · Frozen backbone and representation model; trainable residual adapters. Manuscript source ↗

PAPER UPDATE · 06 / 06 · TRAINING & INFERENCE

Train on Behavior; Preserve Source Content

Supervise the decoded clean prediction with ordered behavior and source-preservation signals.

13

ORDERED BEHAVIOR READER $γ_{j,k} = sim(e_J(V̂_{0|k}), e_J(V_j)).$ $ℓ_act,k = 1 − γ_{+,k} − λ_disc log [exp(γ_{+,k}/τ) / Σ_{j∈K} exp(γ_{j,k}/τ)].$

+ Demonstration− Source− Reversed− Shuffled

The frozen V-JEPA2 reader differentiates through generated videos. Demonstration and control descriptors are detached.

Source preservation

Compare patch sets and appearance statistics. Remove the gradient component opposing behavior, then cap preservation RMS at ¼ of the behavior-gradient RMS without amplifying it.

Training

1,508 real source–target pairs; real-target and offline-generated demonstrations at 3:1. Action-code warm-up precedes behavior supervision.

Inference on OmniBench

Source + instruction → Qwen3-VL rewrites a T2V prompt.
Frozen Bernini generates the demonstration.
Extract $K$ once and run the conditional sampler.

No real target is read; inference needs neither behavior-reader gradients nor per-example optimization.

§3.3 + §4.1 / Appendix D · Warm-up followed by behavior supervision; inference uses a self-generated demonstration. Manuscript source ↗

EXPERIMENTS · QUANTITATIVE CONTEXT

OmniEdit-Bench Temporal: Paper Results

Accuracy-gated scores from the manuscript · 0–100, higher is better · category counts in parentheses

14

Method	Motion semantic (30)	Motion attribute (50)	Temporal composition (70)	Action (150)	Camera (50)	All (200)
VACE	6.80	7.54	11.48	9.23	8.00	8.92
ReCo	5.59	3.14	6.46	5.18	16.43	7.99
OmniVideo2	32.58	13.15	12.64	16.80	14.77	16.29
Kiwi-Edit	29.31	21.70	19.31	22.11	15.39	20.43
LiveEdit	18.80	13.92	10.79	13.44	7.32	11.91
Bernini · frozen	31.36	21.54	13.46	19.73	42.99	25.55
VADD · ours	34.93	24.96	17.64	23.54	46.16	29.19
Direct-FM · reference	38.23	14.56	15.85	19.90	25.82	21.38

ACTION

+3.81 over Bernini

23.54 vs. 19.73 across 150 non-camera cases.

ALL TEMPORAL

+3.64 over Bernini

29.19 vs. 25.55 across all 200 cases.

READING THE NEXT TWO CASES

Selected qualitative examples

High-score cases are a showcase; the table summarizes the complete evaluation.

§4, Omni Temporal table · Direct-FM is a paired-target reference and is not ranked. The following archive scores use Qwen3-VL-32B / 12 frames. Manuscript source ↗

QUALITATIVE · OMNIBENCH

Delete the flying kiss

Delete the action of the girl giving a flying kiss, while maintaining the continuity of the video naturally.

15

**OmniVideo2 · 1.3B**Score 32.0Open video ↗

**LiveEdit · 1.3B**Score 0.0Open video ↗

**ReCo · 1.3B + LoRA**Score 0.0Open video ↗

Clips start together at their original speed. Shorter clips hold their last frame; use the full video link for individual controls.

Additional videosFrozen Bernini · 1.3B ↗

QUALITATIVE · OMNIBENCH

Make the man laugh happily

Make the man laugh happily.

16

**OmniVideo2 · 1.3B**Score 86.0Open video ↗

**LiveEdit · 1.3B**Score 30.8Open video ↗

Clips start together at their original speed. Shorter clips hold their last frame; use the full video link for individual controls.

Additional videosFrozen Bernini · 1.3B ↗

QUALITATIVE · SYNTHETIC

Push forward → Push right

Keep the source block appearance while following the demonstrated trajectory.

17

Clips start together at their original speed. Shorter clips hold their last frame; use the full video link for individual controls.

Appearance × trajectory

Preserve source colors while following the reference motion. Compare the approach, direction changes, and final placement at 1× speed.

View sampled frames and audit records

Time →Frame 0Frame 27Frame 53Frame 80

Source

Reference

Ours

Expected · audit only

Archived source record

Archived self-reconstruction

Uses the manuscript’s author-confirmed my/source pairing. The paper’s provenance note records the older source archive as incorrect. Expected frames are audit-only. Pairing record ↗

QUALITATIVE · SYNTHETIC

Push forward-left → Push back-right

Keep the source block appearance while following the demonstrated trajectory.

18

Clips start together at their original speed. Shorter clips hold their last frame; use the full video link for individual controls.

Appearance × trajectory

Preserve source colors while following the reference motion. Compare the approach, direction changes, and final placement at 1× speed.

View sampled frames and audit records

Time →Frame 0Frame 27Frame 53Frame 80

Source

Reference

Ours

Expected · audit only

Archived source record

Archived self-reconstruction

Uses the manuscript’s author-confirmed my/source pairing. The paper’s provenance note records the older source archive as incorrect. Expected frames are audit-only. Pairing record ↗

QUALITATIVE · REAL

Holding a camera → Looking through it

Edit the action so that the woman raises her silver camera with both hands, bringing it up to her eye to look through the viewfinder.

19

Clips start together at their original speed. Shorter clips hold their last frame; use the full video link for individual controls.

Additional videosFrozen Bernini · 1.3B ↗Real target · reference ↗

Paired-real-target flow-matching variant; retained as labeled in the manuscript figure provenance. Asset provenance ↗

QUALITATIVE · REAL

Static shot → Camera cranes up

Edit the action so that the camera cranes up, moving away from the couple and revealing a wide view of the sunlit forest and valley.

20

Clips start together at their original speed. Shorter clips hold their last frame; use the full video link for individual controls.

Additional videosFrozen Bernini · 1.3B ↗Real target · reference ↗

Paired-real-target flow-matching variant; retained as labeled in the manuscript figure provenance. Asset provenance ↗

QUALITATIVE · REAL

Posing → Walking out of frame

Edit the action so that the smiling woman in a bikini walks from left to right past the camera by a pool, exiting the frame.

21

Clips start together at their original speed. Shorter clips hold their last frame; use the full video link for individual controls.

Additional videosFrozen Bernini · 1.3B ↗Real target · reference ↗

Paired-real-target flow-matching variant; retained as labeled in the manuscript figure provenance. Asset provenance ↗

QUALITATIVE · REAL

Watching → Helping draw

Edit the action so that the mother helps her young daughter draw with colored pencils in a sketchbook at a wooden table.

22

Clips start together at their original speed. Shorter clips hold their last frame; use the full video link for individual controls.

Additional videosFrozen Bernini · 1.3B ↗Real target · reference ↗

Paired-real-target flow-matching variant; retained as labeled in the manuscript figure provenance. Asset provenance ↗

QUALITATIVE · REAL

Typing → Stretching the back

Edit the action so that the woman working on a laptop at her desk suddenly experiences back pain and tries to stretch it out.

23

Clips start together at their original speed. Shorter clips hold their last frame; use the full video link for individual controls.

Additional videosFrozen Bernini · 1.3B ↗Real target · reference ↗

Paired-real-target flow-matching variant; retained as labeled in the manuscript figure provenance. Asset provenance ↗

QUALITATIVE · REAL

Using a phone → Drinking from a mug

Edit the action so that the man wearing glasses looks at his smartphone and takes a drink from a white mug while sitting at his desk.

24

Clips start together at their original speed. Shorter clips hold their last frame; use the full video link for individual controls.

Additional videosFrozen Bernini · 1.3B ↗Real target · reference ↗

Paired-real-target flow-matching variant; retained as labeled in the manuscript figure provenance. Asset provenance ↗

VADD · DISCUSSION

From the Paper to the Evidence

A controlled theory, a practical editor, and visible motion transfer.

25

THEORY

Invariance + information retention

Independent appearance resampling and temporal observability identify the action block at population optimum.

METHOD

Demonstrate, encode, execute

Learn ordered action anchors, couple them to the source through early adapters, and supervise decoded behavior alongside preservation.

EVIDENCE

Keep the experimental tracks explicit

Controlled synthetic outputs, generated-reference Zebra, and the FM variant supply complementary qualitative evidence.

Paper update ↑Experiments ↑Motivation ↑Full Omni gallery ↗