Verba icon indicating copy to clipboard operation
Verba copied to clipboard

Published 20 hours ago • verified publisher icon weaviate

Mass Import Files?

Open Raichuu41 opened this issue 7 months ago • 7 comments

Description

Question/Discussion: What is the best way to mass-import many files? I need to import about 200.000 text files. Currently my only working solution would be to upload all the files in batches of size 500 into github folders. And then import these folders via the GitHub reader one by one manually, whenever the current import is completed. Is there an easier way to do this, possibly directly by sending the file bytes via an API endpoint?

Is this a bug or a feature?

  • [ ] Bug
  • [x] Feature

Steps to Reproduce

[see above]

Additional context

[None]

Raichuu41 avatar Jul 03 '24 19:07 Raichuu41