0
Fork 0
mirror of https://github.com/zeronsh/comet.git synced 2026-11-16 18:57:19 +00:00
comet/crates/visual-gpu/examples
Wing 8a0fded891
Native visuals: agents show interactive Luau visuals rendered by GPUI (#900)
* Render agent visuals natively from sandboxed Luau scripts

show_visual now takes a Luau script instead of an HTML page. The script
builds its data and state once and returns a view function whose ui.*
tree (layout, text, stats, tables, charts, tabs, toggles, sliders,
buttons, links, 2D canvases, 3D scenes) Zeron renders with its own GPUI
elements, so a visual scrolls, clips, blurs under the glass composer and
takes input like any transcript row on every platform.

zeron-visual is the GPUI-free runtime: a Luau sandbox with no I/O, a
memory cap and per-call time budgets, seeded randomness, strict props
whose style rules are the API (theme color tokens, the transcript's type
scale, no fills or borders), canvas drawing commands, and 3D scene
projection. The engine runs every script through check() before showing
it and returns line-numbered errors to the agent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Render visual scenes and shaders on the GPU

- Scenes rasterize with a depth buffer (no more painter's-algorithm
  occlusion errors), clip at the near plane, shade curved solids
  smoothly, and paint as one image; the software path renders in
  parallel row bands at full resolution.
- With a GPU (wgpu, new crate zeron-visual-gpu), scenes render with PBR
  materials (roughness, metal, glow, glass with ior and dispersion),
  soft shadows, MSAA, and bloom; the software renderer is the fallback.
- ui.shader runs Shadertoy-style GLSL on the GPU: iTime, iResolution,
  iMouse, iCamera (orbit), named inputs bound to state, iNoise, and up to
  four buffer passes read as iChannel0-3 with feedback. Composited by
  blend (light by default, so black is clear; alpha; opaque) with
  optional bloom. Each pass is validated with naga so errors reach the
  agent with their line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add visual assets and iOS visuals, and harden untrusted visual input

Assets: show_visual takes files (glTF models, PNG/JPEG/WebP/HDR images)
from the agent's working directory. The engine checks and stores them
beside the script; scenes gain s:model, textures, environment maps and
sprites, and shaders get iTexture0-3. Limits (16 files, 8 MB each, 10 MB
together) fit the MCP server's one WebSocket message to the engine.

Rendering: GPU scenes and shaders fade out toward their edges; plain
translucent surfaces blend unbent (only glass refracts) and, with sprites,
draw before glass so it refracts them; bloom composites correctly in
alpha mode; shader images sample upright as on Shadertoy.

iOS: each visual gets a Rust worker thread in zeron-mobile that runs the
script, lays the tree out, draws canvases and charts, and renders scenes
and shaders with wgpu; the transcript takes each visual's measured
height, and VisualView only paints what arrives and forwards touches.
The demo gains an "Agent visuals" chat (ZERON_DEMO_VISUALS can point it
at a desktop's visuals).

Hardening (security and code review):
- glTF: images stay encoded and are size-checked when drawn; accessor
  shapes are validated; node walks are bounded; parsing can't panic.
- Images: a pixel budget besides the side limit; textures larger than
  the GPU allows are refused.
- wgpu errors are caught or logged, never a panic; the texture cache is
  keyed by content, bounded, and decodes outside its lock.
- The engine checks for a running turn first and removes partial
  writes; asset names may not differ only by case; the MCP server reads
  only regular files, bounded.
- Desktop: released visuals free their GPU images; iTime and iMouse
  follow the docs; no second render when a drag ends.
- iOS: fonts are shared, not leaked per visual; offscreen and
  backgrounded visuals pause; commands see the tree the last one left;
  failed loads retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Render desktop visual frames off the main thread and reuse GPU targets

Desktop scenes and shaders rendered and read back inside GPUI's paint,
blocking the main thread on the GPU every animated frame, so a heavy
agent shader could stall the app. Paint now does only the cheap parts
(input, uniforms, the scene's budgeted Lua draw) and hands the render
to a background thread, one frame in flight per visual; the view shows
the newest finished frame and repaints when the next lands.

Scenes also allocated ~70 MB of render targets every frame. They now
live in a SceneState kept per visual (desktop and iOS) and are remade
only when the size changes; scenes and shaders keep their readback
staging buffer too. The test room's steady frame went from ~8 ms to
~3 ms at 1440x720.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: wing-anara <wing@anara.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
History 2026-10-14 18:04:16 -07:00
..
render_shader.rs Native visuals: agents show interactive Luau visuals rendered by GPUI (#900) 2026-10-14 18:04:16 -07:00