langchaingo icon indicating copy to clipboard operation
langchaingo copied to clipboard

textsplitter: add option to join table rows

Open corani opened this issue 1 year ago • 0 comments

Originally the MarkdownTextSplitter would split tables into chunks for each row (producing a header + single row) in each chunk. This change adds an option to join multiple rows into a single chunk.

Fixes: #938

PR Checklist

  • [x] Read the Contributing documentation.
  • [x] Read the Code of conduct documentation.
  • [x] Name your Pull Request title clearly, concisely, and prefixed with the name of the primarily affected package you changed according to Good commit messages (such as memory: add interfaces for X, Y or util: add whizzbang helpers).
  • [x] Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
  • [x] Provide a description in this PR that addresses what the PR is solving, or reference the issue that it solves (e.g. Fixes #123).
  • [x] Describes the source of new concepts.
  • [x] References existing implementations as appropriate.
  • [x] Contains test coverage for new functions.
  • [x] Passes all golangci-lint checks.

corani avatar Aug 06 '24 09:08 corani