0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
mastra/stores
Arthur Fiorette 464810dcf8
fix(pg): keep PgVector startup warmup from caching a dropped index (#30578)
## Description

A `PgVector` instance no longer caches what its startup warmup read for
an index that `deleteIndex()` or an index rebuild invalidated while the
warmup was running. Before this change, two things could go wrong. The
next `upsert()` or `query()` on that index could fail with `Vector table
"<name>" does not exist`, even after the table had been created again. A
later `createIndex()` could also skip its work and keep the index's
previous metric.

The test `PgVector > Namespace isolation > lazy migration without
createIndex > invalidates readiness after deletion and raw recreation`
failed intermittently in CI with:

```
Error: Vector table "test_namespace_lazy_migration" does not exist
 ❯ PgVector.describeIndexMetadata src/vector/index.ts:1822:27   (MASTRA_VECTOR_PG_DESCRIBE_INDEX_FAILED)
Caused by: Error: Vector table "test_namespace_lazy_migration" does not exist
 ❯ PgVector.describeIndexMetadata src/vector/index.ts:1745:15
```

### Cause

The constructor starts a cache warmup and does not await it
(`stores/pg/src/vector/index.ts`, `cacheWarmupPromise`). The warmup
calls `listIndexes()`, then `getIndexMetadata()` for each listed table,
and that call stores its in-flight promise in `indexMetadataCache`. Then
it writes the index's cache key to `createdIndexes`. The failing
sequence in CI:

1. `new PgVector(...)`: the warmup's `listIndexes()` reads the catalog
and sees the table.
2. `deleteIndex()` drops the table and clears `indexMetadataCache`.
3. `listIndexes()` returns its earlier result. The warmup calls
`getIndexMetadata()`, which caches a new `describeIndexMetadata()`
promise for the dropped table.
4. That read runs before the table exists again and fails at
`describeIndexMetadata` (`:1745`).
5. The table is created again, and `upsert()` calls `getIndexMetadata()`
while the warmup's promise is still in the cache, so it gets the
warmup's rejection (`:1822`).

A second sequence uses the same warmup:

1. The warmup reads the metadata of a cosine HNSW index.
2. `deleteIndex()` clears the caches.
3. `createIndex()` recreates the index with the euclidean metric.
4. The warmup finishes and writes the cosine cache key to
`createdIndexes`.
5. A later `createIndex({ metric: 'cosine' })` matches that key, takes
the cached fast path and leaves the euclidean index in place.

### Changes

- The constructor creates a set of invalidated index names for the
warmup. While the warmup runs, `invalidateIndexCaches()` (called by
`deleteIndex()` and by the index rebuild in `setupIndex()`) adds the
index to it.
- The warmup skips an index in that set before it reads its metadata,
and again before it writes `createdIndexes` and `indexVectorTypes`. That
index is read on its first real use instead.
- The set is released once every metadata read of the warmup has
settled, whether the warmup succeeded or failed. `cacheWarmupPromise`,
which `disconnect()` awaits, now also waits for those reads.

### Known limits

- Only operations on the same instance invalidate its caches. When
another instance or process drops or rebuilds a table, this instance's
warmup, or any metadata read it has in flight, can still cache what it
read. This PR does not change that.
- `setupIndex()` invalidates the caches after `DROP INDEX` and before it
creates the new index, and nothing invalidates them afterwards. A
`query()` that reads the metadata in that window caches the index as
`flat` with the default `cosine` metric until the next invalidation.
Separately, `buildIndex()` with other parameters doesn't update
`createdIndexes`, which still holds the key `createIndex()` stored. A
later `createIndex()` with the original parameters then takes the cached
fast path and leaves the rebuilt index as it is. Both exist before this
PR, which does not change them.
- A warmup that finishes during a rebuild with no `deleteIndex()` before
it can still leave a stale key. `createIndex()` with other parameters
stores its key and then awaits catalog reads in `setupIndex()`. A warmup
that finishes during that wait writes the old key, and the rebuild that
follows invalidates the metadata caches without storing the key of the
index it built. A later `createIndex()` with the old parameters then
takes the cached fast path and leaves the rebuilt index as it is. Here
the warmup writes before the invalidation, so the invalidated set can't
stop it. The same sequence fails before this PR, and it shares its cause
with the `buildIndex()` case above: a rebuild never stores the key of
what it built.

### Tests

There are two new tests in `stores/pg/src/vector/index.test.ts`. Each
first checks that the warmup's listing saw the table, and holds the
listing or one of the warmup's catalog queries with a deferred promise.

- `upserts after deletion and raw recreation while the startup warmup is
still running` holds the warmup's `listIndexes()` until `deleteIndex()`
has run. It then holds the warmup's metadata read of the dropped table
(if one starts) until `upsert()` has been called on the recreated table,
and asserts the upserted row is stored. On the code before this change
it fails every run with the error above.
- `rebuilds with the requested metric after deletion while the startup
warmup is still running` holds the warmup after it has read a cosine
HNSW index's definition. It runs `deleteIndex()` and `createIndex()`
with the euclidean metric, lets the warmup finish, then calls
`createIndex()` with the cosine metric and asserts through
`describeIndex()` that the index is cosine. Without the second check it
fails every run with `metric: 'euclidean'`.

The existing test is unchanged. The vector test file passed 5 runs with
the CI vitest config.

## Related issue(s)

None

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [ ] Test update

## Checklist

- [ ] I have linked the related issue(s) in the description above
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

**ELI5:** When a database index is deleted and rebuilt while the app is
starting, the app can save the old index details in its cache. This
change stops the startup check from saving those old details, so later
saves and searches use the current index.

## Summary

This change fixes stale cache entries in `PgVector` startup warmup. The
constructor starts a background warmup. The warmup lists indexes and
reads each index's metadata. If `deleteIndex()` or an index rebuild runs
during that time, the warmup can cache metadata for an index that
changed or no longer exists.

The change in `stores/pg/src/vector/index.ts` does the following:

- Adds a `warmupInvalidations` field, typed `Set<string> | null`. Warmup
creates a new set when it starts. The field is set to `null` when warmup
finishes.
- `invalidateIndexCaches()` adds the index name to this set when the set
is not `null`.
- Warmup skips an index that is already in the set before it reads
metadata.
- Warmup checks the set again before it writes `createdIndexes` and
`indexVectorTypes`.
- Warmup collects the per-index read promises. Its `finally` block waits
for all of them with `Promise.allSettled`. Then it sets the field to
`null`.

The `finally` block keeps tracking active until every started read
settles. A single rejected read does not end tracking early. The
`cacheWarmupPromise` assignment is not changed in this diff. Its async
body now waits for the reads, so `disconnect()`, which waits for
`cacheWarmupPromise`, also waits for those reads.

Expected effects, as stated in the changeset:

- `upsert()` and `query()` do not fail with "Vector table ... does not
exist" after an index is deleted and created again during startup.
- `createIndex()` does not keep the previous metric when an index is
deleted and recreated, or rebuilt, during startup.

Invalidation applies only to operations on the same `PgVector` instance.
Changes made by another instance or process are not covered.

The change adds a patch changeset for `@mastra/pg` that describes both
fixes.

The PR also adds two tests to `stores/pg/src/vector/index.test.ts`.
According to the file summary, one covers upsert after delete and
recreate during warmup. The other covers metric and index type after
delete, recreate, and rebuild during warmup. The test file contents were
not inspected in this review. No test results were available.

**Scope:** 3 files changed, 132 lines added, 2 lines removed.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
History 2026-11-16 18:40:43 -08:00
..
_test-utils feat(core, pg, libsql, server): run evented workflows and durable agents on storage v2 (#29370) 2026-11-16 08:23:06 -08:00
astra docs: point store READMEs at the new integrations vector pages (#28970) 2026-10-30 11:31:23 +01:00
azure-ai-search chore(deps): update build tools (major) (#29026) 2026-11-03 20:18:21 +01:00
chroma docs: point store READMEs at the new integrations vector pages (#28970) 2026-10-30 11:31:23 +01:00
clickhouse chore: version packages (alpha) (#30261) 2026-11-13 19:58:20 -08:00
cloudflare chore: version packages (alpha) (#30261) 2026-11-13 19:58:20 -08:00
cloudflare-d1 chore: version packages (alpha) (#30261) 2026-11-13 19:58:20 -08:00
convex chore: version packages (alpha) (#30516) 2026-11-16 23:44:41 +01:00
couchbase chore: version packages (alpha) (#30430) 2026-11-16 14:09:31 +01:00
dsql chore: version packages (alpha) (#30261) 2026-11-13 19:58:20 -08:00
duckdb chore: version - exit prerelease mode 2026-11-12 21:12:29 +00:00
dynamodb chore: version packages (alpha) (#30261) 2026-11-13 19:58:20 -08:00
elasticsearch docs: point store READMEs at the new integrations vector pages (#28970) 2026-10-30 11:31:23 +01:00
lance chore: version - exit prerelease mode 2026-11-12 21:12:29 +00:00
libsql chore: version packages (alpha) (#30516) 2026-11-16 23:44:41 +01:00
mongodb test(mongodb): pin store-test MongoDB back to 7.0.45 so it starts on CI runners (#30597) 2026-11-16 16:02:59 -07:00
mssql chore: version packages (alpha) (#30261) 2026-11-13 19:58:20 -08:00
mysql chore: version packages (alpha) (#30261) 2026-11-13 19:58:20 -08:00
opensearch docs: point store READMEs at the new integrations vector pages (#28970) 2026-10-30 11:31:23 +01:00
oracledb chore: version packages (alpha) (#30261) 2026-11-13 19:58:20 -08:00
pg fix(pg): keep PgVector startup warmup from caching a dropped index (#30578) 2026-11-16 18:40:43 -08:00
pinecone chore: version packages (alpha) (#30430) 2026-11-16 14:09:31 +01:00
qdrant docs: point store READMEs at the new integrations vector pages (#28970) 2026-10-30 11:31:23 +01:00
redis chore: version packages (alpha) (#30261) 2026-11-13 19:58:20 -08:00
s3vectors chore(deps): update build tools (major) (#29026) 2026-11-03 20:18:21 +01:00
spanner chore: version packages (alpha) (#30261) 2026-11-13 19:58:20 -08:00
turbopuffer chore(deps): update build tools (major) (#29026) 2026-11-03 20:18:21 +01:00
turso chore: version packages (alpha) (#30516) 2026-11-16 23:44:41 +01:00
upstash chore: version packages (alpha) (#30261) 2026-11-13 19:58:20 -08:00
valkey chore: version packages (alpha) (#30261) 2026-11-13 19:58:20 -08:00
vectorize chore: version - exit prerelease mode 2026-11-12 21:12:29 +00:00
weaviate docs: point store READMEs at the new integrations vector pages (#28970) 2026-10-30 11:31:23 +01:00
oxlint.config.ts refactor(lint): replace ESLint with type-aware oxlint (#27476) 2026-10-18 10:38:35 -07:00