localGPT icon indicating copy to clipboard operation
localGPT copied to clipboard

feat(ingest): load files concurrently

Open Tchekda opened this issue 1 year ago • 0 comments

Currently, all files are loaded in a single thread.

This PR, based on this article, will allow loading the documents in multiple threads which speeds up the process when having a lot of documents.

By default, I have set 8 threads as this is a standard value, but it can be changed in constants.py.

Closes #76

Tchekda avatar Jun 09 '23 09:06 Tchekda