text-splitter icon indicating copy to clipboard operation
text-splitter copied to clipboard

Non-greedy split by semantic level

Open benbrandt opened this issue 8 months ago • 0 comments

Discussed in https://github.com/benbrandt/text-splitter/discussions/226

Originally posted by noau June 12, 2024 Thanks for your great work! I want to know that if it's possible to just split strings on a given semantic level instead of splitting greedy and only stops when the chunk exceeds some given size limits. For example, the two sentences above would be splitted into just

  1. "Thanks for your great work!"
  2. "I want to know that if it's possible to just split strings on a given semantic level instead of splitting greedily and stops only when the chunk exceeds some given size limits."

on a sentence level, ignoring the size limits.

benbrandt avatar Jun 20 '24 09:06 benbrandt