mirror of
https://github.com/mastra-ai/mastra.git
synced 2026-11-17 08:44:46 +00:00
fix(core): recover evented agent runs after worker death (#28054)
Fixes COR-1469. Postgres-backed evented agents now send distributed worker orchestration through the shared broker, allowing Redis to reclaim unfinished work after a worker dies while the original API process remains alive. Other workflow locality and cross-process finish delivery are preserved. Postgres atomically checks step occurrence identifiers before writing results. Interrupted occurrences can be taken over, completed loop iterations reuse committed results and decisions, and stale deliveries cannot overwrite newer results. Regression coverage includes saved-result continuation, nested loop completion, retry publication, state preservation, and legitimate next iterations. Validated SIGTERM and SIGKILL with both a warm surviving worker and a replacement worker: all four cases pass, including five repeated matrix runs (20/20), with the original API caller completing without restart. Core validation passed in two groups (344 and 519 tests); Postgres workflow/run-fencing validation passed 42 tests earlier, and the final focused rerun passed 22 tests. TypeScript, touched-file type-aware lint, formatting, and whitespace checks pass. A Postgres query-plan assertion failed once during the final broad rerun and passed on the focused rerun. The Redis suite is not fully green: after building missing fixture dependencies, the seven affected files passed all 25 tests. The remaining agent-thread Lambda-death signal-drain test times out at agent-thread-cross-process.test.ts:389; removing all core changes and rebuilding reproduces the identical failure on the unchanged baseline. This limitation is not suppressed or changed by this PR. Upgrade @mastra/core and @mastra/pg together and drain older runs first. Interrupted external effects must be idempotent. Resume, individual foreach item guards, time travel, restart/redelivery races, other storage adapters, Redis persistence, and payload-size redesign remain out of scope. Tony’s fencing/recovery sign-off is required before merging and before platform#3224 lands. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## ELI5 If a worker stops during an agent run, another worker can take over while the original API process stays alive. The system protects work that already finished, so recovery does not overwrite completed steps or repeat completed loop iterations. ## What changed - Routed eligible Postgres-backed evented workflow work through the shared broker so another worker can reclaim it. Workflow finish events still cross processes. - Added workflow-step guards and Postgres checks to reject stale or duplicate results. - Reused saved step results, loop decisions, and state during recovery. - Kept legacy step records without occurrence metadata on the existing unguarded path until a guarded write stamps them. - Added recovery tests, Redis Streams takeover fixtures, documentation, and coordinated upgrade guidance. ## Validation - SIGTERM and SIGKILL recovery passed with warm and replacement workers: 20/20, as reported. - Core validation passed in groups of 344 and 519 tests. Postgres workflow and run-fencing validation passed 42 tests earlier and 22 in the focused rerun, as reported. - TypeScript, touched-file type-aware lint, formatting, and whitespace checks passed, as reported. - A Postgres query-plan assertion failed in a broad rerun and passed in a focused rerun. - Redis validation remains partial. Seven affected files passed 25 tests. - The timeout at `agent-thread-cross-process.test.ts:389` also occurred on the unchanged baseline, as reported. ## Upgrade requirements and limitations - Upgrade `@mastra/core` and `@mastra/pg` together. - Drain runs started on older versions before upgrading. Older runs can finish after an upgrade, but crash recovery is not guaranteed. - Make interrupted external effects idempotent. - Resume, foreach-item guards, time travel, restart/redelivery races, other storage adapters, Redis persistence, and payload-size redesign are out of scope. - Tony’s fencing and recovery sign-off is required before merging and before `platform#3224` lands. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: NikAiyer <nick.aiyer@gmail.com> Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com> Co-authored-by: Abhi Aiyer <abhiaiyer91@gmail.com>
This commit is contained in:
parent
8cce2b4019
commit
11c83d5333
27 changed files with 2131 additions and 126 deletions
3
pnpm-lock.yaml
generated
3
pnpm-lock.yaml
generated
Loading…
Add table
Add a link
Reference in a new issue