mirror of
https://github.com/mastra-ai/mastra.git
synced 2026-11-17 08:44:46 +00:00
Character Chunking fix (#2940)
This PR solves an issue with character chunking where having overlap led
to tiny chunks at the end.
Example:
// Before: Created tiny chunks at the end
const text = "ABCDEFGHIJ"; // 10 characters
const chunks = await doc.chunk({
strategy: "character",
size: 4, // chunk size of 4
overlap: 2 // overlap of 2
});
// Old behavior - created tiny chunks:
chunks = [
"ABCD",
"CDEF",
"EFGH",
"GH",
"H",
"I",
"J"
]
// After fix: Maintains proper chunk sizes
chunks = [
"ABCD",
"CDEF",
"EFGH",
"GHIJ"
]
This commit is contained in:
parent
9ba1e97e70
commit
b195f6e1fb
3 changed files with 145 additions and 26 deletions
Loading…
Add table
Add a link
Reference in a new issue