e2eAIOK icon indicating copy to clipboard operation
e2eAIOK copied to clipboard

Refactor doc_loader.py to load documents concurrently using Ray actors or Spark tasks, instead of loading them all at once and then putting them into a dataset

Open chaojun-zhang opened this issue 6 months ago • 0 comments

Refactor doc_loader.py to load documents concurrently using Ray actors or Spark tasks, instead of loading them all at once and then putting them into a dataset

chaojun-zhang avatar Dec 26 '23 06:12 chaojun-zhang