Ben Brandt
Ben Brandt
**Motivation:** Right now, the chunking uses a greedy algorithm. The following would output the following chunks: ```rust let text = "Sentence 1. Sentence 2. Sentence 3. Sentence 4."; splitter.chunks(text, 36)...
With the introduction of the `SemanticSplit` trait, the logic of composing the chunks is now separated from the individual implementations of semantic levels. This should make it much easier to...
One benefit of having the extra Markdown structure, other than having better split points, is we can provide extra context to a given chunk from the headings that are relevant...
### Discussed in https://github.com/benbrandt/text-splitter/discussions/226 Originally posted by **noau** June 12, 2024 Thanks for your great work! I want to know that if it's possible to just split strings on a...
Highlights are that all wasm-tools packages and wasmtime are at their latest versions.
Hi everyone. Looking at the LLM WIT World in Spin, there are some extra properties and parameters that would be great to see in there. Overall, I think what you...