0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00

Character Chunking fix (#2940)

This PR solves an issue with character chunking where having overlap led
to tiny chunks at the end.

Example:

// Before: Created tiny chunks at the end
const text = "ABCDEFGHIJ";  // 10 characters
const chunks = await doc.chunk({
  strategy: "character",
  size: 4,        // chunk size of 4
  overlap: 2      // overlap of 2
});

// Old behavior - created tiny chunks:
chunks = [
  "ABCD",
  "CDEF",
  "EFGH", 
  "GH",  
  "H", 
  "I",  
  "J"   
]

// After fix: Maintains proper chunk sizes
chunks = [
  "ABCD",
  "CDEF", 
  "EFGH",  
  "GHIJ"  
]
This commit is contained in:
NikAiyer 2025-03-13 09:44:27 -05:00 • committed by GitHub
commit b195f6e1fb
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 145 additions and 26 deletions