mirror of
https://github.com/TanStack/ai.git
synced 2026-11-18 19:34:58 +00:00
Add first-class video understanding to the Gemini text adapter. A video content part with `metadata.processing: 'agentic'` routes the turn through the Interactions API for multi-pass agentic understanding (GA on gemini-3.7-flash, gemini-3.6-flash, and gemini-3.5-flash-lite via a new `agentic_video` capability); omit it for the default single-pass generateContent sampling. - Extend GeminiVideoMetadata with processing/fps/startOffset/endOffset and thread fps/offsets into generateContent's videoMetadata. - Route processing:'agentic' video parts through interactions.create, wrapping parts in a user_input step and re-emitting the result as AG-UI chunks. - Add uploadGeminiFile() (Files API upload + poll-until-ACTIVE) and geminiVideoPart() helpers. - Unit test for the routing, an e2e video-understanding feature, docs, a testing/panel demo, and a changeset. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
History
|
|
|---|---|---|
| .. | ||
| public | ||
| skills | ||
| src | ||
| tests | ||
| package.json | ||
| playwright.config.ts | ||
| README.md | ||
| tsconfig.json | ||
| vite.config.ts | ||
Stream Processor Test Panel
A visual testing tool for validating the TanStack AI stream processor.
Features
- Drop Zone: Drag & drop JSON trace files to test
- Sample Traces: Pre-loaded examples from unit tests
- Step-through Mode: Process chunks one at a time
- Side-by-side View: See raw chunks and parsed output
Usage
# From workspace root
pnpm install
cd testing/panel
pnpm dev
Then open http://localhost:3010
Creating Trace Files
Trace files are automatically created when you use the chat interface with a traceId. The api.chat route wraps the adapter with wrapAdapterForRecording, which writes a ChunkRecording (see src/lib/recording.ts) to a file when the stream completes.
You can also create trace files programmatically by subscribing to aiEventClient events with createEventRecording, which reconstructs the same ChunkRecording (AG-UI StreamChunks) from the devtools event stream:
import { createEventRecording } from '@/lib/recording'
// Create a recording instance that listens to aiEventClient events
const recording = createEventRecording('tmp/my-trace.json', 'my-trace-id')
// When done, clean up
recording.stop()
Either path captures:
- Stream chunks (content, tool calls, tool results, done, errors, thinking)
- Final accumulated content
- Tool calls and their results
- Finish reason
Trace Format
{
"version": "1.0",
"timestamp": 1234567890,
"chunks": [
{
"chunk": { "type": "content", "delta": "Hello", ... },
"timestamp": 1234567891,
"index": 0
}
],
"result": {
"content": "Hello world",
"toolCalls": [],
"finishReason": "stop"
}
}