0
Fork 0
mirror of https://github.com/TanStack/ai.git synced 2026-11-18 19:34:58 +00:00
ai/testing/panel
Jack Herrington 49642217ca
feat(gemini): agentic video understanding (GA) (#1294)
Add first-class video understanding to the Gemini text adapter. A video content
part with `metadata.processing: 'agentic'` routes the turn through the
Interactions API for multi-pass agentic understanding (GA on gemini-3.7-flash,
gemini-3.6-flash, and gemini-3.5-flash-lite via a new `agentic_video`
capability); omit it for the default single-pass generateContent sampling.

- Extend GeminiVideoMetadata with processing/fps/startOffset/endOffset and
  thread fps/offsets into generateContent's videoMetadata.
- Route processing:'agentic' video parts through interactions.create, wrapping
  parts in a user_input step and re-emitting the result as AG-UI chunks.
- Add uploadGeminiFile() (Files API upload + poll-until-ACTIVE) and
  geminiVideoPart() helpers.
- Unit test for the routing, an e2e video-understanding feature, docs, a
  testing/panel demo, and a changeset.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
History 2026-09-02 08:15:22 +02:00
..
public updating guitar images 2025-12-03 20:04:13 -08:00
skills feat(ai-skills): add @tanstack/ai-skills — portable Agent Skills middleware (#1236) 2026-08-27 18:33:30 +02:00
src feat(gemini): agentic video understanding (GA) (#1294) 2026-09-02 08:15:22 +02:00
tests ci: type-check examples/** and testing/** to catch call-site regressions (#820) (#857) 2026-07-08 07:16:36 +10:00
package.json feat(ui): add typed headless createChatUI adapters (#1252) 2026-09-01 15:58:34 +02:00
playwright.config.ts AG-UI Support (#244) 2026-01-29 13:50:00 -08:00
README.md ci: type-check examples/** and testing/** to catch call-site regressions (#820) (#857) 2026-07-08 07:16:36 +10:00
tsconfig.json Chunk processing overhaul (#61) 2025-12-03 07:50:43 -08:00
vite.config.ts chore: migrate toolchain to oxc + TypeScript 7 + vite 8 (#949) 2026-07-21 15:00:16 +02:00

Stream Processor Test Panel

A visual testing tool for validating the TanStack AI stream processor.

Features

  • Drop Zone: Drag & drop JSON trace files to test
  • Sample Traces: Pre-loaded examples from unit tests
  • Step-through Mode: Process chunks one at a time
  • Side-by-side View: See raw chunks and parsed output

Usage

# From workspace root
pnpm install
cd testing/panel
pnpm dev

Then open http://localhost:3010

Creating Trace Files

Trace files are automatically created when you use the chat interface with a traceId. The api.chat route wraps the adapter with wrapAdapterForRecording, which writes a ChunkRecording (see src/lib/recording.ts) to a file when the stream completes.

You can also create trace files programmatically by subscribing to aiEventClient events with createEventRecording, which reconstructs the same ChunkRecording (AG-UI StreamChunks) from the devtools event stream:

import { createEventRecording } from '@/lib/recording'

// Create a recording instance that listens to aiEventClient events
const recording = createEventRecording('tmp/my-trace.json', 'my-trace-id')

// When done, clean up
recording.stop()

Either path captures:

  • Stream chunks (content, tool calls, tool results, done, errors, thinking)
  • Final accumulated content
  • Tool calls and their results
  • Finish reason

Trace Format

{
  "version": "1.0",
  "timestamp": 1234567890,
  "chunks": [
    {
      "chunk": { "type": "content", "delta": "Hello", ... },
      "timestamp": 1234567891,
      "index": 0
    }
  ],
  "result": {
    "content": "Hello world",
    "toolCalls": [],
    "finishReason": "stop"
  }
}