mirror of
https://github.com/mastra-ai/mastra.git
synced 2026-11-17 08:44:46 +00:00
A long-running Factory server ("shipyard") grows from about 2 GB to
about 7 GB of memory over six hours, and garbage collection rises from
13% to about 42% of its busy CPU time, with event-loop stalls of up to 2
s. Its top allocation site is the agent controller SSE stream. Every
`display_state_changed` event was `structuredClone`d once for each
connected client in `toWireDisplayState`, then JSON-serialized again
once for each client by the server adapter. `structuredClone` copies
every string, so the current message text and every completed tool
result were duplicated in every snapshot, for every client, on every
update. Any frame waiting in a slow client's queue held its own full
copy.
This PR changes only `@mastra/server`. Nothing changes in `@mastra/core`
or the wire format.
**One snapshot per update.** The session bus hands every stream the same
event object in one synchronous dispatch. The stream handler now builds
one snapshot for that event and gives it to every stream of the session.
A single slot keyed by event identity is cleared in a microtask, so a
held event that is emitted again later gets a fresh snapshot.
**Strings are shared, not copied.** `snapshotForWire` copies the state
the way `JSON.stringify` would read it at that moment. It copies every
object and array, because the reducer updates tool status and results,
input buffers, file operations and subagent progress in place. It
resolves `toJSON` (Dates become ISO strings, as on the wire). It shares
strings and other primitives, which can't change. Shared and circular
references are kept the same way `structuredClone` kept them.
**Serialized once.** When a second stream takes the same frame,
`shareStreamChunk` marks it, and `serializeStreamChunk` caches its JSON
in a `WeakMap`. Every client of the session reuses one string. A lone
client never creates a cache entry.
With no clients connected the bus already does no snapshot work, so
`SessionBus` and its ordering guarantees (a pending coalesced snapshot
is flushed before the next non-coalescible event) are untouched.
**Test plan**
New tests in `agent-controller.test.ts` run through a real session bus,
the real stream route and the real `redactStreamChunk` +
`serializeStreamChunk`, then parse the bytes back.
A slow client with every frame queued before it reads still sees each
step of the turn as it was when emitted. For example, a tool is
`running` before it is `completed`, input-buffer text grows, and file
operations go from 1 to 2. Two clients of one session get identical
frames, and a client of another session sees only its own state.
Re-emitting a held `display_state_changed` event sends the current
state, not the cached frame. A parsed tool result with an own
`__proto__` key arrives unchanged.
Each test fails when the matching piece is broken on purpose: a shallow
copy, a never-cleared slot, a frame shared across events, or plain
assignment of `__proto__`. The first three tests also pass on the old
`structuredClone` code, so they check what clients receive, not how it
is built.
Suites run:
- `packages/server`: full suite 3574 passed. `tsc --noEmit`, lint and
prettier are clean.
- `packages/core`: `display-state.test.ts` 106 passed, including the
coalesced-flush ordering test.
- `server-adapters/hono`: 3524 tests passed. Four files didn't load
locally because `@mastra/client-js` and `@mastra/duckdb` weren't built.
`packages/server/scripts/bench-agent-controller-stream.ts` drives a real
session and the real stream route through one turn. The turn is 500 text
deltas spaced past the 16 ms coalescing window, plus one 100 KB tool
result, giving 1018 frames per client. Each client redacts, serializes,
frames and encodes every frame the way the adapters do. `allocated`
comes from one heap-sampling run and counts collected objects. The other
columns are medians of 7 runs, against `origin/main`:
| clients | allocated | time inside `session.emit` | CPU | GC |
| --- | --- | --- | --- | --- |
| 0 | 1.7 → 1.6 MB | 11 → 16 ms | 48 → 62 ms | 3× / 0.7 ms → 2× / 0.9 ms
|
| 1 | 174.8 → 122.0 MB | 73 → 62 ms | 301 → 341 ms | 7× / 9.7 ms → 14× /
10.1 ms |
| 5 | 851.7 → 349.2 MB | 112 → 50 ms | 1320 → 756 ms | 22× / 39.3 ms →
10× / 19.3 ms |
The benchmark hashes every client's full byte stream. The hash is
`aa4b36c9b3ae0e21` before and after, for both 1 and 5 clients. A
separate one-off run with awkward values was also byte-identical. The
values were Dates, nested Map/Set, shared and circular references,
BigInt, `NaN`, `-0`, sparse and typed arrays, class instances,
integer-like keys, U+2028 and lone surrogates.
For one client, allocation drops by about 30%. The 0-client row runs no
changed code, so its differences are noise. CPU and GC moved both ways
across runs: over about 8 paired runs CPU was 293–416 ms before and
281–390 ms after, and GC time stays flat, so I'd call it neutral. The
gain grows with every extra client on a session.
**Decision needed.** Values that `structuredClone` mishandles now
serialize the same way JSON serializes them, which is how every other
stream event is already sent:
- A function or symbol anywhere in display state used to make
`structuredClone` throw. The stream listener then unsubscribed and the
client silently stopped getting events. Now the value is dropped and the
stream keeps going.
- A class instance with `toJSON` used to arrive as its own fields, for
example `{"cents":150}`. It now sends its `toJSON` output, for example
`"$1.50"`.
- An `Error` with custom enumerable fields used to arrive as `{}`. It
now keeps those fields, for example `{"statusCode":403}`.
The run engine turns tool errors into message strings, so I don't expect
the last two in practice. Please confirm this is acceptable.
**Open risks and questions**
Sharing depends on a convention: once a frame is marked shared it must
not be mutated. No adapter mutates chunks (`redactStreamChunk` returns
`display_state_changed` as-is), and the frames are not frozen. A custom
adapter that mutated a chunk would leak the change to other clients of
the session. In-process consumers of the route also now receive the same
frame object as the other streams of that session.
If one client drains a shared frame quickly and another is stalled, the
cached JSON stays alive until the stalled client reads the frame. Each
queued frame then pins about one serialized frame. That is roughly the
size the old per-client clone held, so not worse, but a stalled client
still costs memory per queued frame.
I haven't reproduced the production growth itself. This PR removes the
per-client clone, the duplicated strings and the repeated stringify. It
doesn't address two other things:
- **Unbounded queues.** The stream handler enqueues without checking
`desiredSize`, so a client that stops reading can make its queue grow
without limit. I haven't checked how each adapter handles a slow socket.
- **Full snapshots.** Every snapshot still carries the full state,
including every completed tool result.
Either could account for part of the 2 → 7 GB growth.
If the handler and an adapter were ever loaded from different builds
(ESM and CJS), they would hold separate caches. Clients would still get
correct bytes, just without the shared serialization. In the current
`dist`, both import `serializeStreamChunk` from the same chunk.
Each adapter still builds and encodes `data: ${json}\n\n` for every
client. That is most of the remaining per-client cost, about 58 MB per
client in the benchmark. Caching encoded frames would mean touching
every adapter, so I left it out.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## ELI5
The server sends the same agent status updates to many viewers. Before
this change, it made a full copy of each update for every viewer. Now it
makes one snapshot per update and reuses the serialized result for
viewers that receive the same update. This uses less memory and less
processing.
## Summary
This change affects only `@mastra/server`. It does not change
`@mastra/core` or the wire format.
The change does the following for `display_state_changed` events:
- Creates one snapshot per dispatched event for all streams in a
session.
- Shares strings and other primitives. Copies objects and arrays.
- Marks a frame as shared when several streams use it. Caches its
serialized JSON so clients reuse one serialization. A single-client
stream does not create a cache entry.
- Clears the snapshot slot in a microtask. A later re-emission of the
same event gets a new snapshot.
The change does not alter `SessionBus` or its ordering guarantees.
## Changes by file
- `.changeset/modern-regions-lead.md` (+5): Adds a patch changeset for
`@mastra/server`.
- `packages/server/src/server/handlers/agent-controller.ts` (+87/-13):
Builds the display-state snapshot. Recursively copies objects and
arrays. Preserves shared and circular references. Resolves `toJSON`.
Unboxes `Number`, `String`, and `Boolean` objects. Clones typed-array
views. Keeps an own `__proto__` key as a data property. Converts
Map-valued fields to records. Reuses one serialized frame for streams
that receive the same event during one synchronous dispatch.
- `packages/server/src/server/server-adapter/serialize.ts` (+23/-2):
Adds `shareStreamChunk<T extends object>(chunk: T): T`, which marks an
object for shared serialization. `serializeStreamChunk` reuses the first
result for marked objects, including failures. Unmarked chunks serialize
on each call. The cache uses weak object keys.
- `packages/server/src/server/handlers/agent-controller.test.ts` (+234):
Adds wire-level tests. The tests cover:
- Queued frames that keep the state from their emission time.
- Identical frames for multiple clients of one session.
- Isolation between sessions.
- Tool results with an own `__proto__` key.
- Boxed primitive values.
- Refreshed state when a held `display_state_changed` event is
re-emitted.
- `packages/server/scripts/bench-agent-controller-stream.ts` (+263):
Adds a benchmark. It drives a session through the stream route with a
set number of clients. It reports CPU time, synchronous `session.emit`
time, GC activity, sampled allocations, wire size, frame count, and
per-client wire hashes.
## Behavior changes
- Clients receive identical bytes for a shared event.
- Message text and tool results are shared, not copied.
- An unclonable value no longer closes the stream. Functions and symbols
are dropped.
- Class instances serialize through `toJSON`.
- Enumerable custom fields on `Error` objects are kept.
- An own `__proto__` key is kept as a data property.
- Boxed `Number`, `String`, and `Boolean` values are unwrapped, as in
JSON serialization.
## Testing (author-reported, not verified here)
- `packages/server`: 3,574 tests passed. `tsc --noEmit`, lint, and
Prettier passed.
- `packages/core` `display-state.test.ts`: 106 tests passed, including
the coalesced-flush ordering test.
- `server-adapters/hono`: 3,524 tests passed. Four files did not load
because `@mastra/client-js` and `@mastra/duckdb` were not built.
- Benchmark, one client: allocation fell from 174.8 MB to 122.0 MB.
- Benchmark, five clients: allocation fell from 851.7 MB to 349.2 MB.
- Full-stream hashes matched before and after for one and five clients.
- CPU and GC results were reported as neutral. The 0-client row runs no
changed code and was reported as noise.
- CI status on the current head: not available.
## Open risks and review questions
- The author has not reproduced the production memory growth.
- Slow-client queues are unbounded.
- Snapshots still include full state.
- Shared frames are not frozen. A custom adapter that mutates a shared
frame could affect other streams in the session.
- Adapters still encode a separate SSE frame per client.
- The author asks reviewers to confirm that the JSON handling for values
that `structuredClone` mishandles is acceptable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: mastracode <284800079+mastra-platform[bot]@users.noreply.github.com>
|
History
|
|
|---|---|---|
| .. | ||
| bench-agent-controller-stream.ts | ||
| check-permissions.ts | ||
| generate-api-cli-route-metadata.ts | ||
| generate-permissions.ts | ||
| generate-route-types.ts | ||
| permission-generator.ts | ||