AgentGarten splits world models in two: code owns the state, a neural renderer draws the pixels

AgentGarten builds training worlds for agents by splitting the job in two: a code engine tracks the state and rules, and a distilled video model turns that state into realistic frames at 36.5 frames per second on a single H100. Pretrained agents dropped into these worlds, with no weight updates and only written notes carried between attempts, discovered shelter-building in hide-and-seek by round 4 and ramp use by round 10.

• 36.5 fps streaming render at 480×832: 438.8 ms per 16-frame block, of which 414.4 ms is the 4-step transformer
• Bitwise-exact training replay: the authors’ block-by-block replay removes a 3.99% relative error in the prior approach, while running 5.4% faster and using 0.2% more memory
• Round 4 / round 10 for hide-and-seek shelters and ramps, versus roughly 25M and 100M episodes in OpenAI’s 2019 self-play RL study (the paper itself says this is not a like-for-like comparison)
• Quarry loader score went from 30 to 90.9 out of 100 in four rounds; a one-lane bridge swap dropped from 71 to 41 seconds

Why split state from pixels

Agent environments have two requirements that usually pull against each other. Simulators are faithful: state is explicit, rules are consistent, and you can inspect or edit anything. But making them look real requires expensive hand-built assets. Video world models look real, but their state is implicit in the pixels. You cannot query where an object is, and a rendering glitch can silently change the world.

AgentGarten’s “code worlds” give each side the job it is good at. A scene program runs in an engine, which updates state and exports a geometry map (depth or surface normals) each step. The neural renderer reads that map plus recent frames and an appearance reference image, and draws the next frames. Information only flows one way: rendering errors cannot feed back into the state. That also means a recorded episode can be re-rendered later in a different style or from a different camera.

The worlds themselves are built by a coding agent, either from a single photo (segment objects, estimate depth and 3D boxes, fit meshes) or from a text description using primitives and engine assets. The coding agent then tests the scene with rollouts and engine checks before it is used.

Making the renderer fast and stable

The renderer starts from Cosmos 3-Nano, adapted to take geometry maps as extra input tokens, then distilled into a 4-step, block-causal model that streams 16 frames at a time. A bounded cache keeps 5 permanent “sink” frames (including the appearance reference) plus the 44 most recent latent frames, so long rollouts do not grow memory without limit.

The more interesting engineering is in the distillation, which the authors call Adversarial Forcing. Standard Self Forcing detaches the encoding of earlier frames, so later losses never teach the model how to remember. Re-running the history with gradients fixes that, but only if the replay matches the original rollout. The usual FlexAttention-based replay drifted by 3.99% relative L2. Replaying block by block with standard attention made it bitwise identical, and slightly faster (2.79 s vs 2.95 s forward). The adversarial part adds a discriminator head on the frozen teacher, with exact gradient-penalty regularization computed via Jacobian-vector products to avoid double backward through fused attention kernels.

The payoff is shown only qualitatively: over 30-second rollouts, the Self Forcing baseline develops repeating textures and loses detail, while the Adversarial Forcing model holds up. There are no quantitative visual-quality metrics.

Agents learn by writing notes, not by training

The agents here are off-the-shelf pretrained models; no agent weights are updated. Each round, an agent gets a task file (goal, available actions, limits, but no solution), sees only rendered first-person frames, and acts by submitting short Python programs. It gets no coordinates, map, or score until the episode ends. Afterward it writes a playbook: what it tried, what it saw, what it doubts, what to test next. The next round starts a fresh conversation with the task file plus all earlier playbooks.

In a one-hider, one-seeker hide-and-seek variant, that loop produced the same emergent strategies OpenAI’s 2019 RL agents found: the hider building cover by round 4, and the seeker using a ramp to get over it by round 10. Four other worlds ran for four rounds each:

Progress was not monotonic. The quarry loader scored 30, then 0, then 30, then 90.9; herding peaked at 90.1 in round 2. In its best run, the loader cleared both rocks, delivered one, and parked with 31 seconds to spare out of 360.

Caveats

• “4 rounds vs millions” is not a sample-efficiency result. The abstract leans on it, but the body concedes these are “fundamentally different learning paradigms.” A large pretrained model that already knows what a ramp is starts from a very different place than an RL policy trained from scratch.
• The agent evaluation is tiny. Four rounds, one episode per round in the four extra worlds, no repeated runs, no variance, and no baseline agent under matched conditions. The agent model itself is not clearly identified in the text.
• Visual quality is shown, not measured. The Adversarial Forcing claims rest on side-by-side figures. The only hard numbers are replay error, latency, and memory.
• Geometry does not carry everything. Depth and normals leave color, material, and object identity unspecified, so those can drift after long occlusions, and downsampled geometry can lose thin structures needed for precise control.
• Someone still has to write the worlds. A coding agent helps, but checking whether a generated world is solvable and meaningful is an open problem, and training compute (32 H100s) is reported without total GPU-hours.

The useful idea for anyone building agent environments is the separation itself. Keep ground truth in code you can inspect, edit, and score, and treat the generative model as a swappable skin on top of it. You get video-model realism without letting the video model decide what is true.