0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
Commit graph

13 commits

Author SHA1 Message Date
shreyas-mastra
655b19993f
chore(security): Vanta 2026-10-30 - zod ^4.6.7, @hono/node-server 2.1.5 (direct bumps) (#28983)
## Vanta 2026-10-30 security fixes (consolidated)

Direct dependency bumps and lockfile updates only. **No
overrides/resolutions added or raised.** The earlier override-based
edits on this branch were reverted; those findings are deferred below.

| Package | From → To | Advisory | Where |
|---|---|---|---|
| zod | ^3.25.76 → ^4.6.7 | zod <4.6.6 (High; no fixed 3.x) |
`client-sdks/react` (dev), `deployers/cloudflare` (dev),
`server-adapters/nestjs` (dev), `packages/editor` (dev),
`packages/playground-ui` (dep), `packages/playground`,
`workflows/_test-utils`, `examples/{durable-agents,inngest}`,
`explorations/longmemeval`,
`observability/_examples/otel-bridge/agent-hub`, e2e fixtures/templates
(`experiment-worker/fixtures/*`, `deployers/template/cloudflare`,
`no-bundling/template`, agent-builder `minimal-mastra-project`) + root
`pnpm-lock.yaml` |
| @hono/node-server | 1.19.x / 2.1.1 → 2.1.5 | CVE-2026-118663 /
GHSA-rmxm-3fg6-px4f (Medium) | direct deps: `examples/agent-builder`,
`otel-bridge/hono-multi/service-{one,two,mastra}`,
`templates/template-customer-refund-agent/client-demo-ui`,
`templates/template-kyc-customer-onboarding`,
`templates/template-security-incident-triage` |

Peer ranges (`^3.25.0 || ^4.0.0`) are unchanged, so consumers are
unaffected. Changeset: patch for `@mastra/playground-ui` (runtime zod
dep).

Supersedes the zod parts of #28075 (that PR also raises a pnpm override
and rewrites the `zod-v3` compat aliases to v4, so it is left as-is for
maintainers).

### Deferred (needs override, upstream release, or is intentional)
- zod v3 compat aliases (`zod-v3` in `packages/core`,
`packages/schema-compat`, `e2e-tests/type-check/template`;
`e2e-tests/client-js/zod-v3`): intentional zod 3 test coverage; no fixed
3.x release. Needs a decision to drop/dismiss.
- zod 3.25.76 transitive via `@arcadeai/arcadejs`, `@mcp-ui/client`,
firecrawl, `agent-browser`, `zod-to-json-schema`, MCP SDK paths:
upstream pins; override-only.
- @hono/node-server 1.19.x transitive via `@modelcontextprotocol/sdk`
(root, `examples/agent`, `packages/mcp/integration-tests`), and npm
lockfiles of `scripts/gh-issue-triage` (existing override, left alone)
and the two templates: override-only.
- graphql-scalars, raw-body, jose (templates/scripts): override-only.
- @azure/msal-node <7.1.0: 6.0.1 via `@azure/identity@4.14.0` (fixed in
4.14.1, blocked by the 1-day `minimumReleaseAge`; lockfile refresh
tomorrow), 5.6.0 via `@microsoft/teams.apps` (no fixed release).
- @opentelemetry/core <2.8.0: via `@google-cloud/pubsub@5` (fix needs
pubsub 6 major).
- @ai-sdk/provider-utils 2.2.8 / ai 4.x: pinned vendored AI SDK v4 line;
no fixed release in that major.
- extract-zip, deepmerge: no patched version.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

This PR updates Zod and `@hono/node-server` versions in selected
packages, examples, and templates to address security advisories. It
also updates playground form code to work with Zod 4.

## Changes

- Updates selected `zod` dependencies to `^4.6.7`.
- Updates selected `@hono/node-server` dependencies to `2.1.5`.
- Updates the playground form’s IP address and record validators for Zod
4.
- Maps a specific Zod validation error to the message `Required`.
- Adds a patch changeset for `@mastra/playground-ui`.

## Validation

Test results are not provided in the supplied change summary.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: shreyas-mastra <327047572+shreyas-mastra@users.noreply.github.com>
2026-10-30 15:30:35 -04:00
Dane - AI (Mastra)
a4620cae04
chore(deps): update formatting & linting (#19388)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: wardpeet <ward@coding-tech.com>
2026-07-20 16:13:28 +02:00
Eric Pinzur
50f5884b41
draft: Add MastraPlatformExporter and MastraStorageExporter, deprecate CloudExporter and DefaultExporter (#16223)
## Description

Renames two built-in observability exporters in `@mastra/observability`
to clearer names that better describe what they do. Both renames use a
**duplicate-and-deprecate** pattern: the original classes are preserved
unchanged (same exporter `name` strings, same error IDs, same runtime
behavior) and marked `@deprecated`; the new classes ship in their own
files with updated identifiers.

| Old | New |
| - | - |
| `CloudExporter` | `MastraPlatformExporter` |
| `DefaultExporter` | `MastraStorageExporter` |

This means existing imports keep working without modification, and any
monitoring rules or dashboards matching the old `CLOUD_EXPORTER_*` error
IDs / `mastra-cloud-observability-exporter` name /
`mastra-default-observability-exporter` name keep firing until users
migrate.

### What changed

- **New classes**:
- `MastraPlatformExporter` in
`observability/mastra/src/exporters/mastra-platform.ts` (uses
`MASTRA_PLATFORM_EXPORTER_*` error IDs and `mastra-platform-exporter`
name).
- `MastraStorageExporter` in
`observability/mastra/src/exporters/mastra-storage.ts` (uses
`mastra-storage-exporter` name).
- Each has its own test file (`mastra-platform.test.ts`,
`mastra-storage.test.ts`).
- **Deprecated classes** in `cloud.ts` and `default.ts` — implementation
byte-identical to the previous release, but now carry `@deprecated`
JSDoc pointing at the new classes. The deprecated reference doc pages
stay at their original URLs
(`/reference/observability/tracing/exporters/cloud-exporter` and
`/.../default-exporter`) so existing bookmarks keep working.
- **Public exports**: all four classes (and their config interfaces) are
exported from `@mastra/observability`.
- **Internal callers** updated to use the new classes: the deprecated
`default` observability config in `default.ts`, the `mastracode` runtime
bootstrap, the `mastra` CLI `init` template, and `@mastra/core` JSDoc
and bootstrap warning messages.
- **Templates and examples**: all 11 templates, the `agent-v6` and
`agent` examples, the `e2e-tests/no-bundling` and `e2e-tests/client-js`
fixtures, and the `examples/inngest` README now use the new classes.
- **Docs**: every overview page, exporter page, processor page,
migration guide, storage reference page, and tracing reference page now
recommends the new classes. Two doc pages were renamed (`cloud.mdx` →
`mastra-platform.mdx`, `default.mdx` → `mastra-storage.mdx`) with
permanent redirects from the old URLs in `docs/vercel.json`. Sidebars
now list both deprecated reference pages alongside the new ones; the
deprecated entries get a `(deprecated)` label and a
`sidebar-item-deprecated` className hook.
- **Codemod**: added `MastraPlatformExporter` and `MastraStorageExporter
→ @mastra/observability` mappings alongside the existing `CloudExporter`
/ `DefaultExporter` entries.

### Trade-off

Each pair of files (`cloud.ts` / `mastra-platform.ts`, `default.ts` /
`mastra-storage.ts`) is ~99% identical today. Bug fixes to the new
classes will not auto-apply to the deprecated ones — but since the
deprecated classes are now frozen for back-compat, that's the intended
outcome until they're removed in a future major.

### Migration

```ts
// Before
import { Observability, DefaultExporter, CloudExporter, SensitiveDataFilter } from '@mastra/observability'
new Observability({
  configs: {
    default: {
      serviceName: 'my-app',
      exporters: [new DefaultExporter(), new CloudExporter()],
      spanOutputProcessors: [new SensitiveDataFilter()],
    },
  },
})

// After
import { Observability, MastraStorageExporter, MastraPlatformExporter, SensitiveDataFilter } from '@mastra/observability'
new Observability({
  configs: {
    default: {
      serviceName: 'my-app',
      exporters: [new MastraStorageExporter(), new MastraPlatformExporter()],
      spanOutputProcessors: [new SensitiveDataFilter()],
    },
  },
})
```

Constructor signatures, environment variables
(`MASTRA_CLOUD_ACCESS_TOKEN`, `MASTRA_PROJECT_ID`,
`MASTRA_CLOUD_TRACES_ENDPOINT`), and runtime behavior are all unchanged.
Update monitoring rules that match on `CLOUD_EXPORTER_*` IDs,
`mastra-cloud-observability-exporter`, or
`mastra-default-observability-exporter` when you migrate — the new
classes use `MASTRA_PLATFORM_EXPORTER_*`, `mastra-platform-exporter`,
and `mastra-storage-exporter`.

## Type of Change

- [x] New feature (non-breaking change that adds functionality)
- [x] Documentation update
- [x] Code refactoring

## Checklist

- [x] Documentation updated (overview, reference, migration guides,
dedicated exporter pages)
- [x] Tests added for the new classes (`mastra-platform.test.ts`,
`mastra-storage.test.ts`); existing `cloud.test.ts` and
`default.test.ts` retained unchanged for the deprecated classes
- [x] Backward compatibility preserved: `CloudExporter` and
`DefaultExporter` keep their original error IDs, exporter `name`, and
runtime behavior

https://claude.ai/code/session_014DDGpY9LbumVTqdWX5aGmr

Co-authored-by: Claude <noreply@anthropic.com>
2026-05-11 15:51:43 +02:00
Roamin
269fbf807b
feat(inngest): migrate to Inngest SDK v4 (#15377)
## Summary

Migrates the `@mastra/inngest` workflow engine from Inngest SDK v3 to
v4.

### SDK Migration
- Update `inngest` from `^3.52.6` to `^4.2.2`
- Remove `@inngest/realtime` (now built into v4)
- Update `createFunction` calls to v4 `triggers` syntax
- Update realtime imports from `@inngest/realtime` to `inngest/realtime`
- Simplify `types.ts` for v4 API compatibility
- Add explicit return types in `workflow.ts`

### Realtime Publish (v4)
- v4 removes the `publish` argument from Inngest function context, so
the previous wiring silently no-op'd at runtime and dropped all realtime
events
- `InngestPubSub.publish()` now uses `inngest.realtime.publish(topicRef,
data)` (the v4 client API), which auto-includes the current runId when
called inside an Inngest function
- Restores `watch` events on `workflow:{workflowId}:{runId}` and
`agent-stream` events on `agent:{runId}`
- Drops the dead `publishFn` constructor argument / `InngestPublishFn`
type

### Polling Performance (~83x faster)
- Replace Inngest API polling with snapshot-based polling in `run.ts`
- Handle `waiting` and `pending` snapshot statuses for sleep steps
- Remove unused `getRuns` method and related imports

### Runtime Correctness
- Fail fast with `NonRetriableError` when workflow storage is
unavailable in `getRunOutput`, `_resume`, and `_timeTravel`
- Guard `_resume` against missing snapshots
- Persist `running` snapshot before events in `timeTravel`/`resume` to
prevent stale reads
- Rollback snapshot on `inngest.send()` failure in `_resume` and
`_timeTravel`
- Fall back to snapshot fields in realtime result shaping when event
payload is incomplete

### Dev-server compatibility
- The v4 SDK's `serve` handler uses a wire protocol that older Inngest
Dev Server releases cannot speak — they respond with `EOF writing
request to SDK` and immediately fail every run.
- Setting `checkpointing: false` (the documented v3-compat toggle) does
**not** restore compatibility.
- Verified working against `inngest/inngest:v1.18.0` and later.
Documented the minimum version in
`docs/src/content/en/guides/deployment/inngest.mdx` and pinned the same
image in `examples/inngest`.

### Docs & Example
- `docs/src/content/en/guides/deployment/inngest.mdx`: drop
`@inngest/realtime` install + `realtimeMiddleware` usage, add the
dev-server compatibility note
- `examples/inngest`: bump `inngest` to `^4`, remove
`@inngest/realtime`, drop the `realtimeMiddleware` wiring, pin the
dev-server docker image to `v1.18.0`

### Docker test infrastructure
- Fix Docker connectivity by passing `serveOrigin` via `registerOptions`
- Update Docker image from `inngest:v1.16.1` to `v1.17.9`

### Test Fixes
- Update assertions for core engine `payload` optimization
(`fmtReturnValue` stripping)
- Fix timeTravel test assertions for `payload`/`input`/`state` fields
- Include `status`/`result`/`state`/`input` from snapshot in polling
results

### Test Results
| Suite | Result |
|-------|--------|
| Unit tests (serve.test.ts) | 10/10 ✓ |
| Basic Workflow Execution | 31/31 ✓ |
| Time Travel + Sleep | 21/21 ✓ |
| Shared Suite (Inngest Engine) | 70/70 ✓ |

Verified end-to-end against an external example project: triggers an
Inngest workflow, subscribes via `inngest/realtime`, and receives the
full `workflow-step-start` → `workflow-step-result` → `workflow-finish`
sequence. Smoke-tested workflows: subscribe, failure,
suspend+resume+sleep, parallel branches, sleepUntil, dountil/foreach
loops, concurrent runs, and synthetic agent-stream channel — all green
against `inngest/inngest:v1.18.0` + Inngest SDK v4.

## ELI5

This PR upgrades the Inngest workflow engine in Mastra from version 3 to
version 4. Inngest is a system that helps you run background jobs and
workflows reliably. The upgrade brings big performance improvements
(polling is now ~83x faster), moves realtime features into the main
library, and fixes the realtime publish path that v4 broke. It also pins
the minimum compatible Inngest Dev Server version, since older releases
don't speak the v4 SDK's wire protocol, and updates the docs and example
app to match.

---------

Co-authored-by: Mastra Code (anthropic/claude-opus-4-6) <noreply@mastra.ai>
2026-05-05 15:33:45 -04:00
alex
2c28a067b8
docs(inngest): use /inngest/api so deployment guide works on core >=1.29 (#15743) (#16186)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 08:46:17 +02:00
Tony Kovanen
f8694b6fa0
feat(core,inngest): durable agent architecture with resumable streams (#12557)
This PR introduces durable agents - a way to make agent execution
resilient to disconnections, crashes, and long-running operations.

## The Problem

Standard agent streaming has two fragility points:

1. **Connection drops** - If a client disconnects mid-stream (network
blip, browser refresh, mobile app backgrounded), all subsequent events
are lost. The client has no way to "catch up" on what they missed.

2. **Long-running operations** - Agent loops with tool calls can take
minutes. Holding an HTTP connection open that long is unreliable. If the
server restarts or the connection times out, the work is lost.

## The Solution

Durable agents address both problems:

**Resumable streams** solve connection drops. Every event is cached with
a sequential index. If a client disconnects at event 5, they can
reconnect and request events starting from index 6. They receive cached
events immediately, then continue with live events as they arrive.

```typescript
// Start streaming
const { runId, output } = await durableAgent.stream('Analyze this data...');

// Client disconnects at event 5...

// Reconnect and resume from where we left off
const { output: resumed } = await durableAgent.observe(runId, { offset: 6 });
// Receives events 6, 7, 8... from cache, then continues with live events
```

**Durable execution** solves long-running operations. Instead of
executing the agent loop directly in the HTTP request, execution happens
in a workflow engine (built-in evented engine or Inngest). The HTTP
request just subscribes to events. If the connection drops, execution
continues. The client can reconnect anytime to observe progress.

## Usage

Wrap any existing `Agent` with durability using factory functions:

```typescript
import { Agent } from '@mastra/core/agent';
import { createDurableAgent } from '@mastra/core/agent/durable';

const agent = new Agent({
  id: 'my-agent',
  model: openai('gpt-4'),
  instructions: 'You are helpful',
});

const durableAgent = createDurableAgent({ agent });
```

**Factory functions for different execution strategies:**

| Factory | Execution | Use Case |
|---------|-----------|----------|
| `createDurableAgent({ agent })` | Local, synchronous | Development,
simple deployments |
| `createEventedAgent({ agent })` | Fire-and-forget via workflow engine
| Long-running operations |
| `createInngestAgent({ agent, inngest })` | Inngest-powered |
Production, distributed systems |

## PubSub and Cache

Durable agents use two infrastructure components:

| Component | Purpose | Default |
|-----------|---------|---------|
| **PubSub** | Real-time event delivery during streaming |
`EventEmitterPubSub` |
| **Cache** | Stores events for replay on reconnection |
`InMemoryServerCache` |

When `stream()` is called, events flow through pubsub in real-time. The
cache stores each event with a sequential index. When `observe()` is
called, missed events replay from cache before continuing with live
events.

**Configure via Mastra instance (recommended):**
```typescript
const mastra = new Mastra({
  cache: new RedisServerCache({ url: 'redis://...' }),
  pubsub: new RedisPubSub({ url: 'redis://...' }),
  agents: {
    // Inherits cache and pubsub from Mastra
    myAgent: createDurableAgent({ agent }),
  },
});
```

**Configure per-agent (overrides Mastra):**
```typescript
const durableAgent = createDurableAgent({
  agent,
  cache: new RedisServerCache({ url: 'redis://...' }),
  pubsub: new RedisPubSub({ url: 'redis://...' }),
});
```

**Disable caching (streams won't be resumable):**
```typescript
const durableAgent = createDurableAgent({ agent, cache: false });
```

For single-instance deployments, the defaults work fine. For
multi-instance deployments (load balancer, horizontal scaling), use
Redis-backed implementations so any instance can serve reconnection
requests.

## Class Hierarchy

- `DurableAgent` extends `Agent` - base class with resumable streams
- `EventedAgent` extends `DurableAgent` - fire-and-forget execution
- `InngestAgent` extends `DurableAgent` - Inngest-powered execution

## Other Additions

- `observe()` method on all durable agents for stream reconnection
- `CachingPubSub` - decorator that wraps any PubSub with automatic event
caching
- Model fallback support with retry logic
- 284 tests covering all durable agent functionality
- Docker-free test setup for InngestAgent

🤖 Generated with [Claude Code](https://claude.ai/code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Taofeeq Oluderu <oluderutaofeeq@gmail.com>
Co-authored-by: Tyler Barnes <tylerdbarnes@gmail.com>
Co-authored-by: Mastra Code (openai/gpt-5.5) <noreply@mastra.ai>
2026-04-30 13:17:05 +03:00
Eric Pinzur
fa8786cc58
deleted non-maintained examples (#15256)
## Description

We haven't maintained these in a long time. Lets remove them

## Related Issue(s)


## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update

## Checklist

- [ ] I have made corresponding changes to the documentation (if
applicable)
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## ELI5

This PR cleans up the repository by deleting a large collection of
example code that hasn't been updated in a while. It's like removing old
instruction manuals from a storage closet to keep things tidy and
prevent confusion—the examples still work, but the team decided they're
not actively maintaining them anymore so it's better to remove them.

## Summary

This PR removes approximately 40+ unmaintained example projects from the
`examples/` directory across multiple categories:

### Agent Examples Removed
- `examples/a2a/` - Agent-to-agent communication example (6 files)
- `examples/auth/` - Authentication example with weather agent (6 files)
- `examples/basics/agents/bird-checker/` - Anthropic-based bird
identification agent
- `examples/basics/agents/hierarchical-multi-agent/` - Supervisor agent
orchestration example
- `examples/basics/agents/multi-agent-workflow/` - Sequential
multi-agent workflow
- `examples/basics/agents/system-prompt/` - OpenAI system prompt example
- `examples/basics/agents/using-a-tool/` - Tool integration example
- `examples/basics/agents/voice-capabilities/` - Voice input/output
agent with PlayAI
- `examples/basics/agents/workflow-as-tools/` - Workflow-based tool
example

### Evaluation (Evals) Examples Removed
Removed 17 evaluation metric examples including:
- Answer relevancy, bias, completeness, content similarity
- Context-related metrics (position, precision, relevancy)
- Contextual recall, faithfulness, hallucination detection
- Keyword coverage, prompt alignment, summarization
- Textual difference, tone consistency, toxicity
- Word inclusion

### RAG (Retrieval-Augmented Generation) Examples Removed
Removed 23 RAG examples including:
- Basic chunking examples (delimiters, size, HTML, JSON, markdown, text)
- Embedding examples (chunk arrays, text chunks, Cohere embeddings)
- Vector store integrations (pgvector, Pinecone, Chroma, LibSQL)
- Advanced RAG patterns (filtering, graph RAG, hybrid search, cleanup,
chain-of-thought)
- Metadata extraction and database-specific configuration

### Scope of Changes
Each example removal includes:
- Environment configuration files (`.env.example`)
- Package manifests (`package.json`)
- TypeScript/JavaScript implementation files (`index.ts`)
- Documentation (README.md)
- Changelog files (CHANGELOG.md)
- Git ignore rules (`.gitignore`)
- TypeScript configuration (`tsconfig.json`)

The PR is purely destructive with no new code additions or replacements.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-10 19:30:14 -07:00
dependabot[bot]
ec27c7db08
chore(deps): bump @hono/node-server from 1.19.8 to 1.19.10 in /examples/inngest (#13814)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-27 14:26:09 +01:00
Grayson Hicks
3b0bd601ec
Update examples to zod v4 (#14238)
## Description

Updated all example projects to use zod v4.3.6, matching the workspace
catalog version. This fixes zod version mismatch issues when examples
are linked to the workspace.

**Changes:**
- Updated 109 example `package.json` files: zod from v3 (`^3.25.76`,
`^3.25.0`, `^3`) to `^4.3.6`
- Updated 109 example `packageManager` fields to
`pnpm@10.29.3+sha512.498e1fb4cca5aa06c1dcf2611e6fafc50972ffe7189998c409e90de74566444298ffe43e6cd2acdc775ba1aa7cc5e092a8b7054c811ba8c5770f84693d33d2dc`
(matching root)
- Fixed 2 zod default imports in `examples/agent-v6` to use named
imports (`import { z } from 'zod'`)
- Updated root `package.json` to include SHA hash for `pnpm@10.29.3`

## Type of Change

- [x] Code refactoring

## Checklist

- [ ] I have made corresponding changes to the documentation (if
applicable)
- [ ] I have added tests that prove my fix is effective or that my
feature works

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated zod dependency to v4.3.6 across all example projects,
addressing compatibility with the latest major version.
* Upgraded pnpm package manager to version 10.29.3 across all projects
for improved performance and bug fixes.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-03-13 08:02:24 -05:00
Lennart
0deea91b27
chore: v1 prep (#12085) 2026-01-20 15:53:55 +01:00
Tony Kovanen
1ed5716830
Real time trace collection for inngest engine (#11973)
## Description

### Durable Observability Spans for Inngest Workflow Engine

This PR implements real-time span export for Mastra's Inngest workflow
engine, replacing the previous batch-at-end approach. Spans are now
durably persisted via `step.run()` memoization, ensuring correct trace
reconstruction across Inngest's replay-based execution model.

**Benefits:**

- Traces appear in real-time as workflows execute, not just at
completion
- Spans survive Inngest crashes/restarts and are correctly reconstructed
on replay
- Nested workflow spans maintain correct parent-child relationships

---

### Key Changes

#### 1. Span Lifecycle Hooks in DefaultExecutionEngine

Added overridable hook methods for span creation/end that subclasses can
implement for durability:

- `createStepSpan()` / `endStepSpan()` / `errorStepSpan()` - for
workflow step spans
- `createChildSpan()` / `endChildSpan()` / `errorChildSpan()` - for
control-flow spans (parallel, conditional, loop)

**Files:** `packages/core/src/workflows/default.ts`,
`packages/core/src/workflows/handlers/step.ts`,
`packages/core/src/workflows/handlers/control-flow.ts`,
`packages/core/src/workflows/handlers/sleep.ts`

#### 2. InngestExecutionEngine Hook Overrides

Overrode the span hooks to wrap operations in `step.run()` for
memoization. Each span operation is persisted on first execution and
replayed from cache on subsequent invocations.

- Uses `executionContext.tracingIds` to maintain trace continuity across
replays
- Passes `tracingOptions` to nested workflows for trace propagation

**Files:** `workflows/inngest/src/execution-engine.ts`,
`workflows/inngest/src/workflow.ts`

#### 3. Span Reconstruction via `rebuildSpan()`

Added `rebuildSpan()` method to `BaseObservabilityInstance` that
reconstructs span objects from exported data, enabling span operations
(`.end()`, `.error()`) to work correctly after Inngest replay. Extended
`CreateSpanOptions` to accept optional `spanId` and `startTime` for
rebuilding.

**Files:** `observability/mastra/src/instances/base.ts`,
`observability/mastra/src/spans/default.ts`,
`packages/core/src/observability/types/tracing.ts`

#### 4. Trace Context Propagation

- Added `tracingIds` to `ExecutionContext` type for maintaining
trace/span IDs across durable operations
- Updated control-flow handlers to propagate trace context to nested
execution contexts
- Fixed `startSpan()` to extract `traceId` and `parentSpanId` from
`tracingOptions` for nested workflow trace joining

**Files:** `packages/core/src/workflows/types.ts`,
`packages/core/src/workflows/handlers/control-flow.ts`

#### 5. Fixed Nested Workflow Span Parenting

Fixed a bug where agent spans inside nested workflows had incorrect
parents. The issue was that wrapped mastra instances were being passed
to nested workflows, causing double-wrapping with stale tracing context.
Now nested workflow steps receive raw mastra, which they wrap fresh with
their own step spans.

**Files:** `packages/core/src/workflows/handlers/step.ts`

#### 6. Deleted SpanCollector

Removed the old `SpanCollector` approach that batched spans at workflow
end.

**Deleted:** `workflows/inngest/src/span-collector.ts`

---

### New Example Workflows

Added parallel nested workflow examples for testing trace propagation:

- `examples/inngest/src/mastra/workflows/inngest-parallel-workflow.ts`
- `examples/inngest/src/mastra/workflows/default-parallel-workflow.ts`
- `examples/inngest/src/mastra/agents/trip-comparison-agent.ts`

These workflows run two city plan workflows in parallel, then compare
results - useful for verifying correct span hierarchies across parallel
nested workflow execution.

## Related Issue(s)

<!-- Link to the issue(s) this PR addresses, using hashtag notation:
Fixes #123 -->

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Code refactoring
- [x] Performance improvement
- [ ] Test update

## Checklist

- [ ] I have made corresponding changes to the documentation (if
applicable)
- [ ] I have added tests that prove my fix is effective or that my
feature works


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Real-time span export: spans are exported immediately on creation and
completion.
* Durable span lifecycle hooks: new APIs to manage span creation,
completion, and error reporting across durable executions.
  * Span reconstruction: rebuild spans from exported data during replay.

* **Bug Fixes**
  * Nested workflow step spans now include output data.

* **Examples**
  * Added a parallel city-trip comparison workflow and agent example.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-01-15 11:27:22 -08:00
Tony Kovanen
39b0a5f334
Inngest observability fixes (#11885)
# Inngest Workflow Tracing Fix Report

## Solution: Deferred Span Collection

Instead of creating spans during execution (which happens on every
Inngest replay), we now:

1. **Collect span metadata during execution** - Using a SpanCollector
class
2. **Create all spans in finalize step** - With proper parent-child
relationships
3. **Use memoization for timing** - Span start times captured via
step.run()

### Architecture

```
During Execution:
  - proxyWorkflowSpan.createChildSpan() returns a CollectorSpan
  - CollectorSpan captures: name, type, entityId, startTime, input, etc.
  - When CollectorSpan.end() is called, it captures: endTime, output, status
  - Span data stored in collector with parent-child hierarchy

In Finalize Step (memoized):
  - Create REAL workflow span first (with memoized IDs)
  - For each collected span: create REAL child span from workflow span
  - Proper hierarchy: workflow → step → conditional → step
  - All spans created with correct parent references
```

## How It Works Now

```
INNGEST ENGINE (AFTER FIX):
  1. Create SpanCollector with trace ID
  2. Create CollectorSpan for workflow (collects metadata)
  3. proxyWorkflowSpan delegates to CollectorSpan
  4. During execution: CollectorSpan collects child span metadata in tree structure
  5. In finalize (memoized - runs only ONCE):
     a. Create REAL workflow span first
     b. Recursively create real child spans from collected data
     c. Each child created via parent.createChildSpan() → proper hierarchy
     d. Look up step results by entityId (step ID)
     e. Use step result timing (startedAt/endedAt) for accurate durations
  6. Result: Proper hierarchical traces with correct timing
```

## Why This Works With Multiple Replicas

A key concern with any Inngest solution is multi-replica safety. Here's
why SpanCollector works correctly even when multiple server replicas are
running:

### Inngest's Replay Model

When an Inngest function runs:
1. The function is invoked from the beginning on every replay
2. `step.run()` calls check if their result is memoized - if yes, they
return instantly
3. Only NEW (not-yet-memoized) steps actually execute
4. The function continues until it hits a new step or completes

### Why SpanCollector Is Multi-Replica Safe

```
Replica A (Invocation 1):
  - Executes step1.run() → actually runs, takes 5 seconds
  - Function suspends, result memoized

Replica B (Invocation 2 - replay):
  - step1.run() → returns instantly (memoized)
  - Executes step2.run() → actually runs, takes 3 seconds
  - Function suspends, result memoized

Replica A (Invocation 3 - replay):
  - step1.run() → returns instantly (memoized)
  - step2.run() → returns instantly (memoized)
  - Executes finalize.run() → creates all spans
  - Function completes
```

**The key insight**: SpanCollector captures the span hierarchy on EVERY
invocation (it's just collecting metadata, very cheap). When finalize
runs (memoized, so only ONCE), it has access to the full collected
hierarchy from that specific invocation.

**No shared state needed** - each invocation:
1. Creates its own SpanCollector (local variable)
2. Collects the same hierarchy (deterministic based on workflow
structure)
3. Only one invocation will execute finalize (Inngest memoization)
4. That invocation creates all the real spans with proper hierarchy

**Timing**: Step results already have memoized `startedAt`/`endedAt`
from when steps actually executed. In finalize, we look up each step's
timing by `entityId` (the step ID) and use that for accurate span
durations.

## Key Insights

1. **Span identity vs span objects**: Inngest's model requires
separating span IDs (memoized) from span objects (created in finalize)

2. **Deferred creation**: By collecting metadata during execution and
creating real spans in finalize, we avoid duplication from Inngest
replays

3. **Tree structure preservation**: CollectorSpan maintains parent-child
relationships during collection, then recreates them with real spans

4. **Multi-replica safe by design**: No shared state across replicas -
each invocation collects the hierarchy independently, and only one will
execute finalize

5. **Agent spans work differently**: Agent/model spans are created
inside step.run() and use normal observability - they don't go through
the collector


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an Inngest example app with weather-driven activity planning, a
planning agent, and local package config.
  * Added Docker Compose for local Inngest development.

* **Bug Fixes**
* Improved observability: traces now collect span hierarchy and accurate
timings, then materialize spans safely (multi-replica safe).

* **Documentation**
* Expanded observability docs and configuration examples for production
exporters.

* **Tests**
  * Added tracing-focused tests to validate observability integration.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Taofeeq Oluder <oluderutaofeeq@gmail.com>
2026-01-15 11:02:06 +02:00
Alex Booker
5af15156b2
docs: move Inngest deployment guide (#11213)
Co-authored-by: Lennart <lekoarts@gmail.com>
2025-12-17 15:21:43 +01:00