0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00

fix(core): recover evented agent runs after worker death (#28054)

Fixes COR-1469. Postgres-backed evented agents now send distributed
worker orchestration through the shared broker, allowing Redis to
reclaim unfinished work after a worker dies while the original API
process remains alive. Other workflow locality and cross-process finish
delivery are preserved.

Postgres atomically checks step occurrence identifiers before writing
results. Interrupted occurrences can be taken over, completed loop
iterations reuse committed results and decisions, and stale deliveries
cannot overwrite newer results. Regression coverage includes
saved-result continuation, nested loop completion, retry publication,
state preservation, and legitimate next iterations.

Validated SIGTERM and SIGKILL with both a warm surviving worker and a
replacement worker: all four cases pass, including five repeated matrix
runs (20/20), with the original API caller completing without restart.
Core validation passed in two groups (344 and 519 tests); Postgres
workflow/run-fencing validation passed 42 tests earlier, and the final
focused rerun passed 22 tests. TypeScript, touched-file type-aware lint,
formatting, and whitespace checks pass. A Postgres query-plan assertion
failed once during the final broad rerun and passed on the focused
rerun.

The Redis suite is not fully green: after building missing fixture
dependencies, the seven affected files passed all 25 tests. The
remaining agent-thread Lambda-death signal-drain test times out at
agent-thread-cross-process.test.ts:389; removing all core changes and
rebuilding reproduces the identical failure on the unchanged baseline.
This limitation is not suppressed or changed by this PR.

Upgrade @mastra/core and @mastra/pg together and drain older runs first.
Interrupted external effects must be idempotent. Resume, individual
foreach item guards, time travel, restart/redelivery races, other
storage adapters, Redis persistence, and payload-size redesign remain
out of scope. Tony’s fencing/recovery sign-off is required before
merging and before platform#3224 lands.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

If a worker stops during an agent run, another worker can take over
while the original API process stays alive. The system protects work
that already finished, so recovery does not overwrite completed steps or
repeat completed loop iterations.

## What changed

- Routed eligible Postgres-backed evented workflow work through the
shared broker so another worker can reclaim it. Workflow finish events
still cross processes.
- Added workflow-step guards and Postgres checks to reject stale or
duplicate results.
- Reused saved step results, loop decisions, and state during recovery.
- Kept legacy step records without occurrence metadata on the existing
unguarded path until a guarded write stamps them.
- Added recovery tests, Redis Streams takeover fixtures, documentation,
and coordinated upgrade guidance.

## Validation

- SIGTERM and SIGKILL recovery passed with warm and replacement workers:
20/20, as reported.
- Core validation passed in groups of 344 and 519 tests. Postgres
workflow and run-fencing validation passed 42 tests earlier and 22 in
the focused rerun, as reported.
- TypeScript, touched-file type-aware lint, formatting, and whitespace
checks passed, as reported.
- A Postgres query-plan assertion failed in a broad rerun and passed in
a focused rerun.
- Redis validation remains partial. Seven affected files passed 25
tests.
- The timeout at `agent-thread-cross-process.test.ts:389` also occurred
on the unchanged baseline, as reported.

## Upgrade requirements and limitations

- Upgrade `@mastra/core` and `@mastra/pg` together.
- Drain runs started on older versions before upgrading. Older runs can
finish after an upgrade, but crash recovery is not guaranteed.
- Make interrupted external effects idempotent.
- Resume, foreach-item guards, time travel, restart/redelivery races,
other storage adapters, Redis persistence, and payload-size redesign are
out of scope.
- Tony’s fencing and recovery sign-off is required before merging and
before `platform#3224` lands.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: NikAiyer <nick.aiyer@gmail.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: Abhi Aiyer <abhiaiyer91@gmail.com>
This commit is contained in:
mastra-platform[bot] 2026-10-23 09:00:13 -05:00 • committed by GitHub
commit 11c83d5333
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
27 changed files with 2131 additions and 126 deletions