0
Fork 0
mirror of https://github.com/TanStack/ai.git synced 2026-11-16 19:38:41 +00:00
🤖 SDK that enhances your applications with AI capabilities
  • TypeScript 99.8%
  • JavaScript 0.1%
Find a file
Tom Beckenham c17bc951ca
feat: add generateWorld() and generateLiveVideo() for prompt-steerable sessions (#1321)
* feat: add generateWorld() activity and Reactor adapter

Add a session-based world generation activity, a first-party
@tanstack/ai-reactor adapter that mints scoped tokens for Orbis
and other Reactor world models, and a ts-react-world example app.

* feat: add reactorVideo adapter for live generateVideo sessions

Mint a scoped Reactor token from generateVideo() for Helios, FastH3,
Orbis, LongLive, and LTX. Live sessions return a token instead of a
download URL, so the browser connects with the js-sdk.

* feat: use BYOK in the ts-react-world example

Paste a Reactor key in the browser. The relay reads x-byok-reactor, then
REACTOR_API_KEY, and mints the session token.

* fix: unlock saved BYOK keys in the world example

Passkey storage locks the keyring after a reload. Show Unlock, treat
locked keys as saved, and call byok.prepare() before minting.

* fix: load Reactor wasm in the world example

Vite was pre-bundling @reactor-team/js-sdk, which broke the relative
import of dist/wasm/reactor_wasm.js. Exclude the SDK from optimizeDeps
and resolve that glue file from the package.

* fix: hydrate BYOK env and passkey peek like ts-react-chat

Call byok.ready() and getEnvKeyStatus() on mount so a refresh shows a
locked passkey key or REACTOR_API_KEY on the relay.

* fix: pre-bundle awaitqueue when the Reactor SDK is unbundled

Excluding @reactor-team/js-sdk from optimizeDeps left its CJS dependency
awaitqueue as a raw named import. Include it so Vite converts AwaitQueue.

* feat: add generateLive() and fold world into the media example

Mint live session tokens with generateLive(). reactorVideo and falLive
(H3 Max Director) use that activity. generateVideo stays the job path.

The media example now has Live and World tabs. The ts-react-world
example is removed.

* fix: play live video and open the key dialog on a missing key

Helios uses set_sr_scale, not set_resolution. Mute the video and
re-attach on unmute so Chrome shows frames. A missing key now
returns byokMissing and opens the header dialog. Seedance is a tab
on the media example. Sherif ignores the dual @fal-ai/client versions.

* fix: proxy fal Director auth and keep Orbis on World

WMA rejects a /tokens/ JWT as Key credentials. The media example opens
H3 Max Director through /api/fal/proxy with the real FAL_KEY. Orbis
stays on generateWorld only. Live and World share seed-image upload.

* ci: apply automated fixes

* fix: return the WMA app id from falLive and lock the proxy

generateLive() now puts fal-ai/minimax-h3-max-director on result.model
so wma(live.model) works. The example proxy only forwards WMA session
paths and Director /ice. Doc snippets type-check under kiira.

* refactor: rename generateLive to generateLiveVideo

The activity, adapter kind, fal factory, and createLive method now use
the LiveVideo name so they match generateLiveVideo().

* ci: apply automated fixes

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
History 2026-09-08 16:23:59 +10:00
.agent/self-learning chore: migrate toolchain to oxc + TypeScript 7 + vite 8 (#949) 2026-07-21 15:00:16 +02:00
.agents/skills docs: consolidate quick starts and sort guides by build path (#1274) 2026-08-31 10:43:31 +02:00
.changeset feat: add generateWorld() and generateLiveVideo() for prompt-steerable sessions (#1321) 2026-09-08 16:23:59 +10:00
.claude docs: consolidate quick starts and sort guides by build path (#1274) 2026-08-31 10:43:31 +02:00
.github feat(ci): start AI review from label and approve waiting checks (#1280) 2026-08-31 21:54:00 +00:00
.grok/skills docs: consolidate quick starts and sort guides by build path (#1274) 2026-08-31 10:43:31 +02:00
.vscode chore: migrate toolchain to oxc + TypeScript 7 + vite 8 (#949) 2026-07-21 15:00:16 +02:00
_artifacts chore: remove stale PHP and Python packages (#643) 2026-05-26 10:06:29 +02:00
agent-scripts/ai-review feat(ci): start AI review from label and approve waiting checks (#1280) 2026-08-31 21:54:00 +00:00
codemods feat(ui): add typed headless createChatUI adapters (#1252) 2026-09-01 15:58:34 +02:00
docs feat: add generateWorld() and generateLiveVideo() for prompt-steerable sessions (#1321) 2026-09-08 16:23:59 +10:00
examples feat: add generateWorld() and generateLiveVideo() for prompt-steerable sessions (#1321) 2026-09-08 16:23:59 +10:00
media docs: use orange award badge (#1046) 2026-08-03 14:57:58 +02:00
packages feat: add generateWorld() and generateLiveVideo() for prompt-steerable sessions (#1321) 2026-09-08 16:23:59 +10:00
patches fix: patch changesets to prevent major bumps from peer deps (#361) 2026-03-10 21:16:25 +01:00
scripts chore: sync model metadata (#1326) 2026-09-05 18:16:02 -07:00
testing feat: add generateWorld() and generateLiveVideo() for prompt-steerable sessions (#1321) 2026-09-08 16:23:59 +10:00
.coderabbit.yaml docs: clarify contribution guidelines (#1237) 2026-08-25 10:05:06 +02:00
.cursorignore chore: migrating to tanstack config (#30) 2025-11-27 12:37:42 -08:00
.editorconfig chore: monorepo setup audit (build, tsconfig, types, docs) (#572) 2026-05-19 15:06:18 +02:00
.gitattributes ci: add more shared TanStack config (#49) 2025-11-29 15:19:41 +11:00
.gitignore ci: add in-process Grok PR review bot (#1277) 2026-08-31 08:40:25 -05:00
.npmrc ci: add more shared TanStack config (#49) 2025-11-29 15:19:41 +11:00
.nvmrc ci: add more shared TanStack config (#49) 2025-11-29 15:19:41 +11:00
.oxfmtrc.json build(adapters): remove unused zod peer dependencies (#1253) 2026-08-27 10:21:32 +00:00
.oxlintrc.json chore: migrate toolchain to oxc + TypeScript 7 + vite 8 (#949) 2026-07-21 15:00:16 +02:00
AGENTS.md docs: add TanStack AI banner to package READMEs (#1316) 2026-09-03 12:33:42 +02:00
CHANGELOG.md chore: remove stale PHP and Python packages (#643) 2026-05-26 10:06:29 +02:00
CLAUDE.md feat: add generateWorld() and generateLiveVideo() for prompt-steerable sessions (#1321) 2026-09-08 16:23:59 +10:00
CODE_OF_CONDUCT.md chore: migrating to tanstack config (#30) 2025-11-27 12:37:42 -08:00
CONTRIBUTING.md docs: add TanStack AI banner to package READMEs (#1316) 2026-09-03 12:33:42 +02:00
kiira.config.ts feat: add generateWorld() and generateLiveVideo() for prompt-steerable sessions (#1321) 2026-09-08 16:23:59 +10:00
knip.json feat(ai-remix): add Remix 3 adapter and guitar chat example (#1289) 2026-09-02 11:10:30 +02:00
LICENSE chore: migrating to tanstack config (#30) 2025-11-27 12:37:42 -08:00
nx.json ci: add in-process Grok PR review bot (#1277) 2026-08-31 08:40:25 -05:00
package.json feat: add generateWorld() and generateLiveVideo() for prompt-steerable sessions (#1321) 2026-09-08 16:23:59 +10:00
pnpm-lock.yaml feat: add generateWorld() and generateLiveVideo() for prompt-steerable sessions (#1321) 2026-09-08 16:23:59 +10:00
pnpm-workspace.yaml feat: add generateWorld() and generateLiveVideo() for prompt-steerable sessions (#1321) 2026-09-08 16:23:59 +10:00
README.md feat: add generateWorld() and generateLiveVideo() for prompt-steerable sessions (#1321) 2026-09-08 16:23:59 +10:00
tsconfig.base.json chore: migrate toolchain to oxc + TypeScript 7 + vite 8 (#949) 2026-07-21 15:00:16 +02:00
tsconfig.docs.json feat: add generateWorld() and generateLiveVideo() for prompt-steerable sessions (#1321) 2026-09-08 16:23:59 +10:00
tsconfig.json chore: monorepo setup audit (build, tsconfig, types, docs) (#572) 2026-05-19 15:06:18 +02:00
vitest.workspace.js ci: add more shared TanStack config (#49) 2025-11-29 15:19:41 +11:00

TanStack AI


TanStack AI

Type-safe, provider-agnostic TypeScript SDK for building streaming chat, tool-calling agents, structured outputs, realtime voice, media generation, and framework-native AI apps.

TanStack AI is built from composable activities and provider adapters. Use one provider or switch between many. Import only chat, or add image, audio, video, speech, transcription, summarization, realtime, Code Mode, devtools, and framework bindings as your app needs them.

Read the docs ->

Start Here

What You Can Build

  • Streaming chat experiences with typed messages, tool calls, reasoning parts, and configurable connection adapters.
  • Type-safe tools that can run on the server or client from one shared toolDefinition() contract.
  • Structured output flows backed by JSON Schema, Zod, ArkType, Valibot, or plain JSON Schema.
  • Multimodal prompts and responses that include text, images, audio, video, and documents.
  • Image, audio, video, speech, transcription, and summarization workflows using a shared generation client pattern.
  • Realtime voice chat with provider adapters for realtime sessions and token minting.
  • Code Mode agents that let an LLM write and execute TypeScript in an isolated sandbox to orchestrate tools with loops, branches, and parallel calls.
  • Devtools and observability pipelines for inspecting messages, tool calls, stream chunks, errors, usage, and OpenTelemetry traces.
  • Framework-native clients for React, Solid, Vue, Svelte, and Preact, plus a headless client for custom runtimes.

Install

Install the core package and the provider/framework packages your app uses:

pnpm add @tanstack/ai @tanstack/ai-openai

For a React chat UI:

pnpm add @tanstack/ai @tanstack/ai-client @tanstack/ai-react @tanstack/ai-openai

OpenRouter is also a good starting point if you want access to many providers through one API key:

pnpm add @tanstack/ai @tanstack/ai-openrouter

Streaming Chat

import { chat, toServerSentEventsResponse } from '@tanstack/ai'
import { openaiText } from '@tanstack/ai-openai'

export async function POST(request: Request) {
  const body = await request.json()

  const stream = chat({
    adapter: openaiText('gpt-5.2'),
    messages: body.messages,
  })

  return toServerSentEventsResponse(stream)
}

Learn more in the Chat & Streaming docs and Connection Adapters docs.

Type-Safe Tools

Define a tool once, then attach a server or client implementation with the same input and output types:

import { toolDefinition } from '@tanstack/ai'
import { z } from 'zod'

const getProducts = toolDefinition({
  name: 'getProducts',
  description: 'Search the product catalog',
  inputSchema: z.object({ query: z.string() }),
  outputSchema: z.array(
    z.object({
      id: z.string(),
      name: z.string(),
    }),
  ),
}).server(async ({ query }) => {
  return db.products.search(query)
})

Learn more in the Tools docs, Tool Approval Flow docs, and Lazy Tool Discovery docs.

Structured Outputs

Use outputSchema when you need typed objects instead of freeform text:

import { chat } from '@tanstack/ai'
import { openaiText } from '@tanstack/ai-openai'
import { z } from 'zod'

const Person = z.object({
  name: z.string(),
  age: z.number(),
})

const person = await chat({
  adapter: openaiText('gpt-5.2'),
  messages: [{ role: 'user', content: 'Ada Lovelace, 36' }],
  outputSchema: Person,
})

Learn more in the Structured Outputs docs.

Media, Realtime, and Code Mode

  • Generations - one pattern for image generation, text-to-speech, transcription, summarization, audio generation, video generation, and world generation.
  • Realtime Voice Chat - build low-latency realtime voice experiences.
  • Code Mode - let models write and execute TypeScript inside a secure isolate.
  • Code Mode with Snippets - give Code Mode reusable runtime capabilities.

Providers

Official adapters include:

Package Use it for
@tanstack/ai-openrouter 300+ models through one OpenRouter API, with per-request cost tracking
@tanstack/ai-openai OpenAI chat, image, video, speech, transcription, realtime, and provider tools
@tanstack/ai-anthropic Anthropic Claude chat, thinking, tools, structured outputs, and Vertex Claude
@tanstack/ai-gemini Google Gemini chat, image, speech, and audio generation
@tanstack/ai-vertex Gemini on Vertex AI with regional endpoints and Google Cloud credentials
@tanstack/ai-ollama Local Ollama models
@tanstack/ai-grok xAI Grok chat, images, and realtime
@tanstack/ai-groq Groq low-latency inference
@tanstack/ai-elevenlabs ElevenLabs realtime voice, speech, transcription, music, and sound effects
@tanstack/ai-byteplus BytePlus Seed chat, Seedance video, Seedream image, and Seed Speech TTS/ASR
@tanstack/ai-fal fal.ai image, video, audio, speech, and transcription models
@tanstack/ai-reactor Reactor live world and video generation (Orbis, Happy Oyster, LingBot, Helios, FastH3)
@tanstack/ai-cloudflare Cloudflare Workers AI chat, embeddings, image, speech, transcription, and AI Gateway

The adapter system is tree-shakeable by activity. Import openaiText for chat, openaiImage for images, falVideo for video, geminiSpeech for TTS, and so on.

Framework Packages

Package What it provides
@tanstack/ai-client Headless chat, realtime, and generation clients
@tanstack/ai-react React hooks including useChat, useRealtimeChat, and generation hooks
@tanstack/ai-solid Solid hooks for chat and generations
@tanstack/ai-vue Vue composables for chat and generations
@tanstack/ai-svelte Svelte 5 factories for chat and generations
@tanstack/ai-preact Preact hooks for chat
@tanstack/ai-react/ui, @tanstack/ai-solid/ui, @tanstack/ai-vue/ui, @tanstack/ai-svelte/ui Headless UI components for chat interfaces. @tanstack/ai-react-ui, @tanstack/ai-solid-ui, and @tanstack/ai-vue-ui are deprecated.

Advanced Docs

  • Middleware - hook into chat configuration, chunks, tool calls, usage, errors, and structured outputs.
  • OpenTelemetry - emit vendor-neutral GenAI traces and metrics.
  • Observability - subscribe to typed TanStack AI events.
  • Per-Model Type Safety - narrow model options and content modalities to the selected model.
  • Runtime Adapter Switching - switch providers at runtime.
  • Tree-Shaking - ship only the activities and adapters you use.
  • Agent Skills - install TanStack AI skills into Claude Code, Cursor, GitHub Copilot, Codex, and other coding agents with TanStack Intent.

Get Involved

Partners

CodeRabbit Cloudflare
AI and you?

We're looking for TanStack AI partners to join our mission. Partner with us to push the boundaries of TanStack AI and build amazing things together.

LET'S CHAT

Explore the TanStack Ecosystem

...and more at TanStack.com.