Support for Text Chunking with Overlap in TokenTextSplitter

[alfredogangemi]

trafficstars

Expected Behavior

The TokenTextSplitter class should support splitting text into chunks with an optional overlap between them. This would enable better context preservation when processing long texts that are chunked for downstream tasks such as embeddings or text analysis.

TokenTextSplitter splitter = TokenTextSplitter.builder().withChunkSize(800).withOverlapSize(50).build();

Current Behavior

Currently, the TokenTextSplitter class splits text into chunks without any overlap. While this works for cases where chunks are fully independent, it may result in loss of context between adjacent chunks, particularly in tasks like embeddings where continuity can improve results.

Context

Why it's needed Chunking with overlap is critical for use cases where preserving context across chunks is necessary. For example:

• Embedding generation, where continuity between chunks improves semantic representations.
• Language models that benefit from adjacent context during token predictions.

Alternatives considered The functionality can be implemented by extending TokenTextSplitter locally. However, a native implementation would ensure consistency and reusability across projects.