GS-Agent

Creating 4D Physical Worlds With Generative Simulation

Hongxin Zhang1,2, Chunru Lin1, Junyan Li1, Zhou Xian2, Tsun-Hsuan Wang2, Chuang Gan1
1University of Massachusetts Amherst   2Genesis AI
arXiv Paper

Physically-grounded 4D output

One Prompt → A Simulatable 4D World

GS-Agent composes a scene in the physics engine and rolls it forward in time. Because the output is a real simulation — not pixels — every frame exposes consistent geometry and particle-level dynamics. Toggle the modality below.

Abstract

Creating dynamic and physically realistic 4D worlds from natural language descriptions is both fascinating and challenging. Traditional computer graphics methods rely on manual creation, requiring extensive human effort to fine-tune materials, motions, and visual fidelity. Recent advances in generative foundation models have sparked interest in learning to generate such 4D worlds from large-scale data; however, existing methods still struggle to ensure physical plausibility and controllability. In this work, we take a different path by leveraging foundation models to construct an agentic system that emulates how humans traditionally create 4D worlds, yet automates the entire process. We present GS-Agent, an end-to-end multi-agent framework that integrates physics engines in the loop to generate realistic, dynamic, and controllable 4D physical worlds from natural language. Inspired by how humans build 4D worlds, GS-Agent decomposes the task into entity management, covering 3D asset curation, material tuning, placement, and motion control, and rendering configuration, including camera and lighting manipulation. Multiple agents with distinct expertise interact with the physics engine via code, seek multimodal feedback, and collaborate to iteratively construct 4D worlds that align with the given descriptions. Experimental results show that GS-Agent effectively converts natural language into diverse and physically plausible 4D worlds exhibiting rich interactions among liquids, deformable objects, and rigid bodies, while achieving cinematic camera and lighting control. We envision GS-Agent as a foundation for a new paradigm in 4D world generation, empowering creative content creation and physical AI.

How it works

A Multi-Agent System Driving a Physics Engine

  • Manager Agent — interprets the natural language description, decomposes it into actionable sub-tasks, and delegates them one at a time to the Entity or Render Agent. It configures global simulation and visualization options (time step, substeps, gravity, solver settings), controls timeline progression by advancing and resetting the scene, verifies intermediate results, and triggers the final video capture.
  • Entity Agent — curates assets by retrieving from a library (BlenderKit and PolyHaven) via a text- and image-embedding index, generating new meshes when retrieval fails, and falling back to primitives. It places entities under spatial constraints, iteratively tunes physically plausible material parameters (e.g., Young's modulus, Poisson's ratio, yield stress) from engine feedback, and controls motion through PD controllers, position/velocity/force commands, and particle emitters.
  • Render Agent — grounds verbal directives into camera placement and intrinsics, scripts camera motion as executable trajectory code (orbits, dollies, entity tracking) synchronized with simulation steps, configures lighting, and provides visual feedback to the other agents.

Agents communicate through a SendMessage tool and interact with the physics engine through structured tool interfaces. All agents generate and refine executable code, request multimodal feedback — physics boundary checks, program runtime information, images, and videos — and iterate until the natural language goals are satisfied. The final 4D world is represented as executable simulation code together with a rendered video.

Comparison

GS-Agent vs. Video & Agent Baselines

Same prompt, five systems. GS-Agent is compared against state-of-the-art text-to-video models (Sora 2, Wan 2.2) and foundation-agent baselines (SWE-Agent and our enhanced SWE-Agent w/ Visual variant) that write simulation code directly. Select a scene:

GS-Agent (Ours)
Sora 2
Wan 2.2
SWE-Agent
SWE-Agent w/ Visual

Interactive & editable

Controllable Worlds

Because a GS-Agent world is a real simulation, it can be steered through natural language. Starting from an initial scene, the user issues follow-up instructions — adjusting rendering effects, creating new entities, and controlling object motion — and GS-Agent re-simulates the world consistently after each turn.

User
User A white ball hitting a blue ball on the floor.
GS-Agent
User
User Replay the collision as a slow-motion close-up.
GS-Agent
User
User Add more billiard balls and rack them into a triangle.
GS-Agent
User
User Speed up the white cue ball so it breaks the rack.
GS-Agent

BibTeX

@misc{zhang2026gsagent,
      title={GS-Agent: Creating 4D Physical Worlds With Generative Simulation},
      author={Hongxin Zhang and Chunru Lin and Junyan Li and Zhou Xian and Tsun-Hsuan Wang and Chuang Gan},
      year={2026},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}