langstream icon indicating copy to clipboard operation
langstream copied to clipboard

Check for bad text-splitter agent config

Open ddieruf opened this issue 1 year ago • 0 comments

It would have been nice to be notified about a bad agent config before attempting deployment

Got a larger chunk overlap (100) than chunk size (3), should be smaller.

attempted this

  - name: "Split into chunks"
    type: "text-splitter"
    output: "output-topic" # optional
    configuration:
      splitter_type: "RecursiveCharacterTextSplitter"
      chunk_size: 3
      separators: ["\n\n", "\n", " ", ""]
      keep_separator: false
      chunk_overlap: 100
      length_function: "cl100k_base"

ddieruf avatar Aug 16 '23 18:08 ddieruf