Research-quality prompts
Central claim: Long-video papers have not converged on a new metric system. Instead, they keep adding drift, repetition, collapse, and VLM-stability patches alongside VBench-style proxies. The real paradigm shift is evaluation of world consistency under intervention.
Hidden assumption: DINO, CLIP, start-to-end differences, or VLM scores are often treated as evidence of long-term state consistency, even though these proxies can mistake stillness, repetition, or superficial similarity for memory.
Failure boundary: The difficult cases are continued state change after the camera leaves, returning to a previously visited location, separating dynamic foreground from static background, and comparing models with incompatible control interfaces.
Connection: Evaluation is expanding from open-loop T2V toward camera and action control, 3D geometry, closed-loop revisits, hidden-state evolution, and downstream utility.
Concrete experiment: Build matched trajectories for the same physical process under continuous visibility and under occlusion or camera look-away. Measure reappearance, state progress, identity, and geometry, then reserve human review for cases where the metrics disagree.
Discussion seed: Should a small team build a broad but shallow capability tree, or a narrow and rigorous diagnostic benchmark for dynamic out-of-sight consistency?
Open questions
- What evidence is strong enough to call a similarity score consistency rather than mere appearance proximity?
- Should static-scene revisit and dynamic out-of-sight evolution share one leaderboard?
- How should VLM-judge prompts, sampling, and model versions enter a reproducible protocol?
- How can T2V and interactive world models share tasks without conflating control ability with world consistency?
- With limited staffing, should the benchmark prioritize dataset scale, model coverage, or evaluation breadth?
- Should human alignment be reported as average correlation or as disagreement structure on difficult examples?