localGPT
localGPT copied to clipboard
feat(ingest): load files concurrently
Currently, all files are loaded in a single thread.
This PR, based on this article, will allow loading the documents in multiple threads which speeds up the process when having a lot of documents.
By default, I have set 8 threads as this is a standard value, but it can be changed in constants.py
.
Closes #76