AG-WebGL

AG-WebGL turns a Three.js canvas into a live surface that AI agents can see, reason about, and modify in real time, transforming the 3D scene itself into the generative UI.
Instead of rendering text or static components, agents stream structured scene mutations directly to a Three.js canvas. When a user describes what they want, “add a glowing sphere on the left, bring in a directional light from above, rotate the cube slowly,” a multi-agent LangGraph pipeline interprets the intent and emits a stream of AG-UI custom events: scene:object_add, scene:object_update, scene:camera_move, scene:light_add, scene:animation_start. Each event hits the frontend through CopilotKit’s useCopilotAction hooks and mutates the live canvas in real time. Objects appear, move, and update as the agent thinks, not after it finishes.
The agents are not blind to the canvas they are editing. A purpose-built scene serializer continuously feeds the agent a hybrid context payload: a full index of every object in the scene, full detail on the most recently touched objects, and a spatial zone map. This means agents can reason about what already exists, avoid collisions, and respond to relative instructions like “move it closer to the red sphere” without the context window growing unbounded as the scene gets complex.
The canvas is the interface. The agent output is never text. It is geometry, light, motion, and camera position, streamed live to a Three.js scene that updates with every event.
Stack: AG-UI · CopilotKit · A2UI · LangGraph · LangChain · Three.js · Next.js · FastAPI