langchainrb
langchainrb copied to clipboard
Add ability to parse code
Something like this is needed so we can load codebases in Vector DBs
@santib This opens up an interesting problem -- how do we chunk code?
@santib This opens up an interesting problem -- how do we chunk code?
Yeah, I just used the Text one for simplicity, but checking https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/code_splitter seems like they get the separators for each language, and that's it.
I can change this PR to do something similar if you want
@santib Yeah, I like that!
Closing due to inactivity.