mirror of
https://github.com/zeronsh/comet.git
synced 2026-11-16 18:57:19 +00:00
* Render agent visuals natively from sandboxed Luau scripts show_visual now takes a Luau script instead of an HTML page. The script builds its data and state once and returns a view function whose ui.* tree (layout, text, stats, tables, charts, tabs, toggles, sliders, buttons, links, 2D canvases, 3D scenes) Zeron renders with its own GPUI elements, so a visual scrolls, clips, blurs under the glass composer and takes input like any transcript row on every platform. zeron-visual is the GPUI-free runtime: a Luau sandbox with no I/O, a memory cap and per-call time budgets, seeded randomness, strict props whose style rules are the API (theme color tokens, the transcript's type scale, no fills or borders), canvas drawing commands, and 3D scene projection. The engine runs every script through check() before showing it and returns line-numbered errors to the agent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Render visual scenes and shaders on the GPU - Scenes rasterize with a depth buffer (no more painter's-algorithm occlusion errors), clip at the near plane, shade curved solids smoothly, and paint as one image; the software path renders in parallel row bands at full resolution. - With a GPU (wgpu, new crate zeron-visual-gpu), scenes render with PBR materials (roughness, metal, glow, glass with ior and dispersion), soft shadows, MSAA, and bloom; the software renderer is the fallback. - ui.shader runs Shadertoy-style GLSL on the GPU: iTime, iResolution, iMouse, iCamera (orbit), named inputs bound to state, iNoise, and up to four buffer passes read as iChannel0-3 with feedback. Composited by blend (light by default, so black is clear; alpha; opaque) with optional bloom. Each pass is validated with naga so errors reach the agent with their line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add visual assets and iOS visuals, and harden untrusted visual input Assets: show_visual takes files (glTF models, PNG/JPEG/WebP/HDR images) from the agent's working directory. The engine checks and stores them beside the script; scenes gain s:model, textures, environment maps and sprites, and shaders get iTexture0-3. Limits (16 files, 8 MB each, 10 MB together) fit the MCP server's one WebSocket message to the engine. Rendering: GPU scenes and shaders fade out toward their edges; plain translucent surfaces blend unbent (only glass refracts) and, with sprites, draw before glass so it refracts them; bloom composites correctly in alpha mode; shader images sample upright as on Shadertoy. iOS: each visual gets a Rust worker thread in zeron-mobile that runs the script, lays the tree out, draws canvases and charts, and renders scenes and shaders with wgpu; the transcript takes each visual's measured height, and VisualView only paints what arrives and forwards touches. The demo gains an "Agent visuals" chat (ZERON_DEMO_VISUALS can point it at a desktop's visuals). Hardening (security and code review): - glTF: images stay encoded and are size-checked when drawn; accessor shapes are validated; node walks are bounded; parsing can't panic. - Images: a pixel budget besides the side limit; textures larger than the GPU allows are refused. - wgpu errors are caught or logged, never a panic; the texture cache is keyed by content, bounded, and decodes outside its lock. - The engine checks for a running turn first and removes partial writes; asset names may not differ only by case; the MCP server reads only regular files, bounded. - Desktop: released visuals free their GPU images; iTime and iMouse follow the docs; no second render when a drag ends. - iOS: fonts are shared, not leaked per visual; offscreen and backgrounded visuals pause; commands see the tree the last one left; failed loads retry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Render desktop visual frames off the main thread and reuse GPU targets Desktop scenes and shaders rendered and read back inside GPUI's paint, blocking the main thread on the GPU every animated frame, so a heavy agent shader could stall the app. Paint now does only the cheap parts (input, uniforms, the scene's budgeted Lua draw) and hands the render to a background thread, one frame in flight per visual; the view shows the newest finished frame and repaints when the next lands. Scenes also allocated ~70 MB of render targets every frame. They now live in a SceneState kept per visual (desktop and iOS) and are remade only when the size changes; scenes and shaders keep their readback staging buffer too. The test room's steady frame went from ~8 ms to ~3 ms at 1440x720. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: wing-anara <wing@anara.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
History
|
|
|---|---|---|
| .. | ||
| mandelbulb.glsl | ||