langstream
langstream copied to clipboard
Check for bad text-splitter agent config
It would have been nice to be notified about a bad agent config before attempting deployment
Got a larger chunk overlap (100) than chunk size (3), should be smaller.
attempted this
- name: "Split into chunks"
type: "text-splitter"
output: "output-topic" # optional
configuration:
splitter_type: "RecursiveCharacterTextSplitter"
chunk_size: 3
separators: ["\n\n", "\n", " ", ""]
keep_separator: false
chunk_overlap: 100
length_function: "cl100k_base"