gbif-dl icon indicating copy to clipboard operation
gbif-dl copied to clipboard

Add support for webdataset

Open faroit opened this issue 3 years ago • 1 comments

Using Webdataset is a great way to speed up the training pipeline and also makes it convenient to share and download achieves of datasets (e.g. by uploading to Zenodo).

Addressing this issue should involve:

  • a method to write webdataset tar files using the gbif_dl.io method.
  • a torch dataset class/pipeline to parse the dataset.

faroit avatar Dec 13 '20 10:12 faroit