0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
mastra/packages/rag/src
mastra-platform[bot] 82c98f9f1c
fix(rag): keep long JSON string content when chunking JSON documents (#30407)
> Opened by @factory-project:7d5175c3-3a20-42ac-aba9-756f183136f4

JSON chunking (`MDocument.chunk({ strategy: 'json' })`) silently
discarded most of a long string value whenever the field's serialized
path was longer than 13 characters, or the value contained characters
that JSON escapes. The splitter budgeted a fixed 20-character overhead
regardless of the real field name, and then dropped every piece that
still exceeded `maxSize`. The result was silent data loss with no error,
no warning, and no config that avoided it — a 3000-character value under
a 16-character key kept 120 characters at `maxSize` 500, and a
14-character key at `maxSize` 21 produced zero chunks.

The fix sizes each piece from the value it is about to emit: it measures
the serialized envelope of the candidate piece and its full field path,
cuts only at code-point boundaries so astral characters are never split,
and never discards a completed piece. When a field path alone leaves no
room for a single character, the remainder is emitted as one oversized
chunk instead of being dropped.

**Result:** Approved — `d8caff7436`

**Checks on this head**

| Check | Result | Tests |
| --- | --- | --- |
| New #30405 regression tests
(`packages/rag/src/document/document.test.ts`) | ✅ Passed | 10 passed |
| Full `packages/rag` suite | ✅ Passed | 471 passed across 24 files |
| `tsc --noEmit` (`packages/rag`) | ✅ Passed | |
| `oxlint --type-aware .` (`packages/rag`) | ✅ Passed | 0 warnings, 0
errors |
| `oxfmt --check` on the changed files | ✅ Passed | |
| New tests on base code (`5b7f1302`) | ✅ Catches the bug | 8 of 10 fail
without the fix |
| Independent round-trip probe (16 cases) | ✅ Passed | base: 9 fail →
head: 16 pass |

**Follow-ups**

- `minSize` remains accepted but ignored for JSON chunking; already
documented in `docs/src/content/en/reference/rag/chunk.mdx`.
- Long strings nested two or more levels deep are still emitted as one
oversized chunk rather than split; this boundary is unchanged by the fix
and already described in the `maxSize` docs.
- `ensureAscii` escaping runs after sizing, so an ASCII-escaped chunk
can exceed `maxSize`; unchanged and out of scope.
- `handlePrimitiveValue` tests "fits" against `{ [key]: value }`
(immediate key only) while emitting with the full path; this predates
the change and cannot lose or oversize data, since a chunk that fits
without the path also fits as emitted.

Adversarial check: I tried to falsify the fix with inputs the author did
not test — a 100k-character value (round-trips, 103 chunks, 11 ms),
astral characters (no chunk cut mid-pair), space-heavy values, nested
and array shapes, and the smallest allowed `maxSize` (21) — and the
round-trip invariant held on every case, while a probe written before
reading the diff reproduced the content loss on the base commit.

Fixes #30405

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## ELI5

JSON chunking could drop parts of long text values. This change keeps
the full text and splits it into smaller pieces. Most pieces now fit the
size limit. A few cases still produce pieces that are larger than the
limit.

## Summary

This change fixes content loss in `MDocument.chunk({ strategy: 'json'
})` for `@mastra/rag`. The splitter reserved a fixed 20-character
overhead for each long string value. Field names or JSON escape
sequences can use more space than that overhead. The splitter then
discarded the pieces that exceeded `maxSize`.

## Changes

- `packages/rag/src/document/transformers/json.ts` (+51/-20):
- `RecursiveJsonTransformer` sizes each candidate piece by its
serialized chunk and full field path. It no longer uses the fixed
20-character overhead.
- Pieces split at code-point boundaries. The splitter prefers a
preceding space.
- If the field path leaves no room for one character, the remaining
string is emitted as one oversized chunk.
  - The filter that removed string chunks above `maxSize` was removed.
- Non-string primitive values that do not fit in a new chunk are
appended without a serialized-size check.
- `packages/rag/src/document/document.test.ts` (+108): Adds JSON
chunking tests. They cover long field names, escaped string values,
surrogate pairs, non-string primitives, and the no-room case. A
1,000,000-character string test checks reconstruction, a 500-character
chunk limit, and completion within 1,000 ms.
- `docs/src/content/en/reference/rag/chunk.mdx` (+1/-1): Documents that
long string values are split without discarding content. It states that
chunks may exceed `maxSize` when the field path leaves no room for one
character, when `ensureAscii` escaping expands a value (the default), or
when a long value is nested in an object.
- `.changeset/smooth-donkeys-split.md` (+5): Adds a patch changeset for
`@mastra/rag`.

## Linked issue

Issue `#30405` reports content loss with `maxSize: 90`, `minSize: 1`,
and `ensureAscii: false`. The issue expects complete reconstruction. It
also expects serialized chunk sizes within `maxSize`. It asks for a
configuration error when the field name leaves no room for a character.

This change does not implement the configuration error. When the field
path leaves no room, it emits the remaining content as one oversized
chunk.

## Limits

- Long strings nested two or more levels deep remain one oversized
chunk.
- `ensureAscii` escaping can still produce a chunk larger than
`maxSize`.
- When the field path leaves no room for one character, the remainder is
emitted as one oversized chunk.

## Verification

The author reports the following results on the PR head:
- 10 new regression tests pass. 8 of them fail on the base commit.
- The `packages/rag` suite passes: 471 tests across 24 files.
- `tsc --noEmit` and `oxlint --type-aware .` pass in `packages/rag`.
`oxfmt --check` passes on changed files.
- A 16-case round-trip probe passes on the PR head. The same probe fails
9 times on the base commit.

These results were not re-run for this summary. CI status on the current
head: not available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: mastra-platform[bot] <mastra-platform[bot]@users.noreply.github.com>
Co-authored-by: mastra-platform[bot] <284800079+mastra-platform[bot]@users.noreply.github.com>
History 2026-11-15 08:44:43 -08:00
..
document fix(rag): keep long JSON string content when chunking JSON documents (#30407) 2026-11-15 08:44:43 -08:00
graph-rag fix(rag): reject nonfinite GraphRAG edge weights (#29956) 2026-11-10 14:38:33 -08:00
rerank fix(rag): reject invalid rerank component weights (#29957) 2026-11-10 14:15:21 -08:00
tools feat: pass the authenticated user to DynamicArgument callbacks (#28612) 2026-11-12 10:55:12 -05:00
utils feat: pass the authenticated user to DynamicArgument callbacks (#28612) 2026-11-12 10:55:12 -05:00
index.ts Save and reload GraphRAG knowledge graphs instead of rebuilding them every run (#21704) 2026-08-17 20:03:16 -07:00