0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
mastra/stores/redis/src
Arthur Fiorette 8559e74767
feat(storage): workflow run summary listings and batched run ownership reads (#30274)
## Description

Adds `summary: true` listings that return only each run's status,
timestamp and parent, and `WorkflowsStorage.getRunOwnerships()` to read
many runs' ownership at once. Nothing in core calls them yet, so no
workflow behavior changes.

### Summary listings

`listWorkflowRuns({ summary: true })` now reduces each run's snapshot
to:

```ts
{ status?, timestamp?, parentWorkflow?: { workflowId, runId } }
```

`parentWorkflow` is present only when the stored parent has both IDs,
and is never `null`. A field the run doesn't have is left out. `context`
and every other snapshot field are never returned. Run fields outside
the snapshot (`runId`, `workflowName`, `resourceId`, `createdAt`,
`updatedAt`) are unchanged, and so is `total`.

| Where the snapshot is reduced | Stores |
|---|---|
| In the query (only the summary fields leave the database) | PostgreSQL
and LibSQL (already projected `{ status, timestamp }`, now add the
parent), MySQL, MSSQL, OracleDB, Spanner, DSQL, Cloudflare D1 and
Durable Objects, MongoDB (`project()`) |
| In the app, after reading the whole snapshot | DynamoDB, Redis,
Valkey, Upstash, Convex, in-memory |

This PR doesn't change the status filter. Among the SQL stores,
PostgreSQL (expression index on `snapshot ->> 'status'`), Spanner
(generated status column) and OracleDB (function index on `JSON_VALUE`)
filter on an index, and LibSQL storage v2 runs have an indexed `status`
column. MySQL, MSSQL, DSQL and Cloudflare D1 and Durable Objects extract
the status from each row's JSON.

MSSQL now caches whether the snapshot table has a `resourceId` column.
The non-summary `resourceId` filter uses the same cache, with the same
result as before.

### Batch ownership reads

`getRunOwnerships({ runIds })` returns a `Map` from run ID to the record
`getRunOwnership()` would return for that ID; runs that were never
claimed are left out. The default reads each run with
`getRunOwnership()`, at most 8 at a time, and stops starting new reads
once one fails. Overrides:

| Store | Round trips |
|---|---|
| PostgreSQL | One query for all IDs (one array parameter) |
| LibSQL | One query per 500 IDs |
| MySQL, MSSQL | One query per 500 IDs, joining the requested IDs so the
column's collation matches them as the single-run read does |
| MongoDB | One `find` per 1,000 IDs |
| DynamoDB | One strongly consistent `BatchGetItem` per 100 IDs,
unprocessed keys retried with backoff; IDs that differ only in case name
one item, as in the single-run read |
| Upstash | One pipeline request per 100 IDs |
| Redis, Valkey | One single-key script call per ID (works across
cluster hash slots), up to 500 in flight |
| In-memory | Reads every record in one pass |
| Convex, DSQL, Spanner, OracleDB, Cloudflare D1 and Durable Objects |
Base default |

### In-memory `updatedAt`

The in-memory store now moves a run's `updatedAt` on
`updateWorkflowResults()`, `updateWorkflowState()` and
`updateWorkflowRunHeader()`, as the database adapters already did. A
write a status guard rejects leaves it alone.

### Conformance

In `@internal/storage-test-utils`, run on every store with run fencing:
the summary listing shape for v1 and storage v2 runs, including a null
parent, a parent with one ID and a missing timestamp; `getRunOwnerships`
over 1,001 IDs with live, released, expired and unclaimed runs, each
record equal to the single-run read, plus IDs that differ from a claimed
one only by a trailing space or case (MySQL, MSSQL and DynamoDB match
those in the single-run read). The `updatedAt` cases run through
`createWorkflowsTests`. Valkey and Cloudflare Durable Objects don't run
that suite: Valkey writes through the same path as Redis, and Durable
Objects don't implement partial writes.

Checks run locally: `pnpm build:core`, core `check`, lint and `tsc` of
every touched store, and the conformance suite against in-memory,
LibSQL, PostgreSQL, MySQL, MSSQL, MongoDB (replica set), Redis, Valkey,
DynamoDB Local and Upstash (serverless-redis-http). MSSQL has 15
failures outside this PR's cases that fail the same way on `main`
locally. DSQL, Convex, Spanner, OracleDB and Cloudflare were not run
locally.

Stack 1 of 4; the recovery pass in the later PRs uses these reads.

## Related issue(s)

Part of #28712

## Type of change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [x] Performance improvement
- [x] Test update

## Checklist

- [x] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

**ELI5:** Workflow runs can now be listed in a smaller form that holds
only the status, time, and parent workflow. The PR also adds a way to
look up the owners of many runs in one call. Storage adapters return
only the fields that callers need.

## Overview

This PR adds two storage capabilities for workflow runs:

- `listWorkflowRuns({ summary: true })` returns a reduced snapshot for
each run. The snapshot contains `status` and `timestamp`. It adds
`parentWorkflow: { workflowId, runId }` only when both parent IDs are
non-empty strings.
- `WorkflowsStorage.getRunOwnerships({ runIds })` returns ownership
records for many runs. It returns a `Map` keyed by run ID and omits
unclaimed runs.

The in-memory store also changes. Successful writes refresh `updatedAt`.
A write rejected by a status guard leaves `updatedAt` unchanged.

The PR description states that core does not call these APIs yet.
Workflow behavior does not change in this PR.

## Changes by area

**Core (`@mastra/core`)**
- `storage/domains/workflows/base.ts`: Adds a default `getRunOwnerships`
implementation. It deduplicates run IDs and reads at most 8 records
concurrently through `getRunOwnership`. It omits unclaimed runs. A
failed read rejects the batch and stops further reads from starting.
- `storage/domains/workflows/inmemory.ts`: Adds `summary` support to
`listWorkflowRuns`. Summary reduction runs after filtering and
pagination. `updateWorkflowResults`, `updateWorkflowState`, and
`updateWorkflowRunHeader` refresh `updatedAt` after successful writes.
- `storage/types.ts`: Documents the fuller reduced-snapshot shape for
`summary`. It changes no runtime behavior.

**Store adapters**

Summary support (`summary: true`) is added in: Cloudflare D1, Cloudflare
Durable Objects, Convex, DSQL, DynamoDB, LibSQL, MongoDB, MSSQL, MySQL,
OracleDB, PostgreSQL, Redis, Spanner, Upstash, and Valkey.

Reduction happens in the database for these stores:
- SQL stores: PostgreSQL, LibSQL, MySQL, MSSQL, OracleDB, Spanner, DSQL
- Cloudflare D1 and Durable Objects
- MongoDB (projection)

Reduction happens in application code for these stores:
- Convex, DynamoDB, Redis, Upstash, Valkey

`getRunOwnerships` is added in: PostgreSQL, LibSQL, MySQL, MSSQL,
MongoDB, DynamoDB, Redis, Upstash, and Valkey. The batch strategy
differs by store:
- PostgreSQL: one query.
- LibSQL, MySQL, MSSQL: chunks of up to 500 IDs.
- MongoDB: batches of 1,000 IDs.
- DynamoDB: strongly consistent batch reads of up to 100 keys.
Unprocessed keys retry with exponential backoff.
- Upstash: pipeline batches of 100.
- Redis and Valkey: one single-key ownership script call per run. Calls
run in batches of up to 500.

Cloudflare D1 and Durable Objects include `resourceId` in summary
queries only when the table has that column. The column check is cached.

**Conformance test utilities (`stores/_test-utils`)**
- `createSummaryListingTests`: Covers summary shapes, parent metadata
handling, and comparison with full listings. It skips stores without
run-fencing support.
- `createUpdatedAtTests`: Covers `updatedAt` on partial writes. For
storage v2 run journals, it checks header and state updates, including
guard-rejected writes.
- Run-fencing tests: Add batch ownership read cases for live, released,
expired, and unclaimed runs. They also cover inputs larger than
MongoDB's 1,000-item chunk size, collation lookalike IDs, and empty
input. The `untilLapsed` helper polls for up to 5 seconds, checking
every 20 ms.

**Changesets**

Patch changesets cover the affected store packages. A minor changeset
covers `@mastra/core`.

## Scope

- Files changed: 30
- Lines added: 1,555
- Lines removed: 52

## Verification

The author reports these local results. They have not been checked
against the current head:

- `pnpm build:core`, core checks, lint, and TypeScript checks for
touched stores were run.
- Conformance runs were run against the listed stores. The author
reports 15 MSSQL failures outside this PR's test cases. The author
reports the same failures on `main`.
- Conformance tests were not run locally for DSQL, Convex, Spanner,
OracleDB, or Cloudflare.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
History 2026-11-13 11:45:22 -03:00
..
storage feat(storage): workflow run summary listings and batched run ownership reads (#30274) 2026-11-13 11:45:22 -03:00
cache.ts Fix atomic pubsub replay writes on Redis Cluster (#30174) 2026-11-12 06:27:56 -08:00
index.test.ts fix(core,redis,valkey): stop duplicate observe frames and save evented suspensions before publishing (#27352) 2026-10-23 12:39:25 -05:00
index.ts fix(core,redis,valkey): cut per-chunk cache round-trips in durable agent streaming (#23161) 2026-09-07 17:57:22 -07:00
integration.test.ts fix(core,redis,valkey): stop duplicate observe frames and save evented suspensions before publishing (#27352) 2026-10-23 12:39:25 -05:00