Multi-Agent Problems in World Models

Scope and background

This reading-group survey focuses on world models connected to video generation and embodied interaction. An active agent makes choices; a manipulated object need not do so, and an additional camera is not an additional agent. Object, particle, and motion-factor representations remain useful related approaches, with these distinctions preserved.

A latent action model infers a compact representation of a transition from before-and-after observations, then uses that representation with the current observation to predict the next one. The representation need not recover a true command or intention. Multiple moving entities raise a further question: can their changes be represented separately?

Recent work provides several answers to parts of the broader problem. The papers below differ in action labels, geometry, state access, environments, and outputs; their reported results do not form a single performance ranking. The survey covers fifteen papers checked through 2026-10-23, with relevant original method and evaluation sections read. It is not a reproduction study.

Six questions and their reading route

Question	Given information	What the model must provide	Reading entry
1. Action learning	Video transitions without per-agent action labels	Separate latent action or motion representations	FLAM; PlaySlot, LPWM; OTF-LAM and GDLAM as background
2. Control binding	Actions or instructions for named subjects	A future video in which the correct subjects execute them	ActionParty; Incantation
3. Other-agent behavior	History and externally specified actions for only part of the scene	The behavior of agents whose actions were not supplied	COMBAT; WorldMind
4. Cooperative planning	Partial observations and candidate actions	Imagined consequences that help select the acting agent's next action	COMBO
5. Shared world and persistence	Multiple views, action conditions, and history	Different videos that agree on shared entities, events, and state changes	ME-World; OverSeer; WorldCast briefly
6. Agent-count generalization	Training with fewer agents than may be present at inference	Usable identities and information exchange for more agents	Gamma-World; Multics

Mechanisms and boundaries

Separate representations. FLAM assigns latent action channels to learned scene factors, while the dynamics can still use the full current scene. PlaySlot adds object-slot actions with discrete prototypes and continuous variation; LPWM models stochastic particle-level transitions. OTF-LAM decomposes local effects but aggregates them into one latent action, and GDLAM learns grouped representations. Neither a factor nor a group is guaranteed to be an autonomous agent.

Known controls. ActionParty links each subject's action and state to spatial locations and predicts video together with subject coordinates. Incantation provides per-entity language action slots. These approaches use action supervision; they are not action-free latent-action discovery.

Unspecified behavior. COMBAT conditions on one player's actions and implicitly generates the opponent's behavior in video. WorldMind explicitly connects state estimation, NPC decisions, action conditions, and generation. Its state and decision interfaces use geometry supervision during training and skill information; not all state is recovered directly from pixels. Implicit behavioral generation is not evidence that a transferable policy has already been trained.

Choosing actions. COMBO builds a visual world estimate from partial observations, predicts partners' actions, imagines joint-action consequences, and searches over its own choices. Different candidates deliberately produce different possible futures. It does not require the planner to directly control every partner.

Agreeing on one world. ME-World jointly generates egocentric streams using shared action conditions and geometry-aligned history. OverSeer adds generated fixed auxiliary views for continuity beyond moving local views; these are not access to a true omniscient state. WorldCast illustrates distributed local generation coordinated through shared state and scene memory. Its reported block-synchronous evaluation is not a guarantee under arbitrary network delays.

More agents. Gamma-World uses symmetric identity encoding from a preset identity pool and shared hub communication, including a reported two-agent-training to four-agent-inference setting. Multics organizes information through player, group, and global levels. Testing more agents does not establish unlimited population size or arbitrary online entry and exit.

The sequence connects research questions rather than implemented systems: ActionParty does not consume FLAM outputs, and COMBO is not a downstream integration of COMBAT or WorldMind.

Research-quality prompts

Central claim: “Multi-agent world modeling” covers distinct tasks. Which agents' actions are given, whether outputs support decisions or multiple viewpoints, and whether agent count changes are essential to interpreting a result.

Hidden assumption: Identity labels, action annotations, coordinates, geometry, skill rules, and history may be available at different stages. Comparing methods without stating these information conditions can obscure the actual difficulty.

Failure boundary: Decomposed latents need not align with agents; controllability need not supply other agents' decisions; cross-view agreement need not imply physical correctness; population generalization remains bounded by the tested setting and representation.

Connection: The literature connects latent dynamics, controllable video generation, embodied planning, and persistent multi-view generation through a common concern: how actions and shared state are represented and checked across interacting entities.

Concrete experiment: A possible public-literature diagnostic is to hold an initial two-character scene and character B's action fixed while changing A's action, then inspect action binding and shared-object updates across two views. A separate condition lets B follow a response policy. This distinguishes misplaced controls from legitimate interaction effects. This is a discussion example; no experiment or result is reported here.

Discussion seed: Which information interface is the bottleneck for a given task: action representation, subject binding, other-agent decisions, shared state, or scalable communication?

Open questions

When do learned action factors track an active agent rather than an object, body part, or local visual change?
How can action-to-identity binding survive occlusion, close interaction, and role or position exchanges?
When is implicit video prediction sufficient for useful other-agent behavior, and when is an explicit state and decision interface needed?
Which prediction errors change cooperative action choices, rather than only the perceived quality of a generated video?
How should cross-view event agreement and off-screen persistence be tested separately from visual similarity?
Which identity and communication structures transfer to more agents, and which depend on a preset population or interaction pattern?

These are discussion questions synthesized from the literature, not established novelty claims or assigned projects. See the reference analyses for the seven core readings and all eight supporting or background entries.