There has been significant progress in teaching machines to model the world. From large language models operating over text, to emerging world models that predict pixels or states over time, we are steadily increasing the scope of what machines can represent and simulate. Yet there remains a missing abstraction—one that sits between raw sensory data and full-scene representations.
That abstraction is the object.
Not as a bounding box, not as a label, but as a neural object: a persistent, structured, and manipulable unit of the world.
The Problem with Pixels-All-the-Way
Most current world models follow a clean formula:
(history + action) → next frame
Feed in what happened. Get back what happens next. It's elegant. It scales. And it's fundamentally limited.
Here's why. Try telling a pixel-based world model: "move the chair 10 cm to the left." You can't. There is no chair in there. There are activations that, collectively, look like a chair when decoded. But the chair has no identity. No boundary. No handle you can grab.
This is like asking a language model to be the program instead of writing code. We already know how that story ends.
Pixel-based world models bury objects inside their weights. They can render a scene, but they can't reason about what's in it. You can't edit one object without risking the rest. You can't reuse a table across two different rooms. You can't track the same cup from the counter to the dishwasher.
Everything that makes real-world interaction possible — persistence, identity, manipulation — is invisible at the pixel level.
The Problem with Full Scenes
So maybe go bigger? Represent the whole scene as a single structure — a NeRF, a Gaussian splat, a large implicit field.
This gives you structure, but at the wrong granularity. A full scene is:
Too entangled to edit locally — move one wall, and you're re-optimizing the whole representation. Too large to reuse — you can't drop "the kitchen from scene 47" into a new environment without hauling everything along. Too monolithic to compose — you can't build new worlds from parts.
If pixels are too fine-grained and scenes are too coarse, then the right abstraction sits exactly where you'd expect: in the middle. At the level of objects.
What Is a Neural Object?
A neural object is a learned representation that encapsulates four things about a single entity: its geometry (shape, volume, how it occupies space), its appearance (texture, material, how it responds to light), its state (where it is, how fast it's moving, whether it's deformed), and its affordances (what you can do with it).
That last one matters more than people think.
A coffee mug is not just a mesh with a brown texture. It affords grasping. It affords containing liquid. Tip it, and the coffee spills. These aren't decorative properties — they're the reason robots need to understand mugs at all.
Affordances can't be cleanly inferred from pixels alone. They can't be represented at the scene level without entangling every object's interaction potential with every other. They belong at the object level. They always have.
Unlike traditional 3D assets, neural objects aren't static files sitting in a library. They're learned from data, conditioned on context (a text description, a reference image, a partial observation), and generative — you can sample new instances, edit existing ones, and compose them into novel configurations.
And unlike pixels, they're factorized. They isolate structure from context. The chair is the chair regardless of the room it's in.
Objects as Interfaces
The real power here isn't just representational. It's interfacial.
Think about what a neural object enables between humans and machines. A human can inspect an object, rotate it, tweak its geometry, swap its material. A model can generate it, predict its future state, simulate what happens when it collides with something else. Both sides are working with the same unit. This creates something rare in AI: a genuinely iterative, bidirectional workflow where human intent and machine capability meet on common ground.
Now think about what it enables between machines and machines. A perception system detects and initializes objects. A planner reasons about their states and spatial relations. A controller executes actions on them. A simulator updates their dynamics. Each subsystem reads and writes to the same abstraction.
Without objects, these subsystems communicate through either raw pixels (inefficient, ambiguous) or shared latent spaces (fragile, non-modular). Neither is great. Neural objects give you a structured, inspectable, interoperable interface — the kind of clean boundary that makes complex systems actually work.
The Factorized World Model
This points toward a different architecture for world models. Instead of one monolithic function that maps history and actions to the next observation, you get a factorized stack:
Objects at the center — the core representation. Relations around them — spatial constraints, physical contacts, semantic groupings. Dynamics governing them — how objects evolve when forces are applied, time passes, or interactions occur. Rendering on top — how objects become the pixels you actually observe.
This is the same move that transformed software engineering. We went from monolithic executables to structured code — functions, classes, modules, interfaces. Neural objects are to world models what functions are to programs: reusable, composable building blocks that make the whole system more than the sum of its parts.
Persistence Changes Everything
Real-world interaction depends on a deceptively simple property: continuity.
The mug you picked up is the same mug you set down. The door you opened five minutes ago is still open. The screwdriver you used on the first bolt is the same one you use on the third.
This sounds trivial. It isn't.
Without explicit object representations, a world model must reconstruct identity from scratch at every timestep. It has to re-discover, from raw sensory patterns, that the blob of pixels in frame 312 corresponds to the same entity as the blob in frame 1. Every frame is an amnesiac.
Neural objects give you a natural handle for identity over time. And once you have persistent identity, you get long-horizon reasoning for free. You get memory. You get causality — the ability to say "this happened because that object was in that state."
Worlds Are Not Generated. They Are Constructed.
Here's where this leads.
If objects are the right abstraction, then worlds become compositions of objects, relations, and rules. Models generate objects, not just pixels. Systems operate over objects, not just frames. Users manipulate objects, not just prompts.
This is a paradigm shift. The current approach says: "throw everything into a giant model and let it hallucinate a plausible future." The object-centric approach says: "build a world from parts, with structure you can inspect, edit, and trust."
One path gives you impressive demos. The other gives you systems you can actually use.
The Hard Questions
I don't want to pretend any of this is solved.
What's the right representation for a neural object — implicit fields, Gaussian splats, meshes, some hybrid nobody's proposed yet? How do you discover object boundaries without supervision? How do you encode affordances and dynamics in a way that generalizes? Where does learned physics end and explicit simulation begin?
And the deepest question: do we need to engineer objects into our models, or will they emerge from scale the way linguistic structure emerged in LLMs?
I don't know. But I believe the field has been avoiding this question for too long, seduced by the elegance of end-to-end pixel prediction. The bitter lesson taught us to scale. It didn't say we should scale without structure.
Objects are how the world is built. They should be how we model it.