gbif-dl
gbif-dl copied to clipboard
Add support for webdataset
Using Webdataset is a great way to speed up the training pipeline and also makes it convenient to share and download achieves of datasets (e.g. by uploading to Zenodo).
Addressing this issue should involve:
- a method to write webdataset
tar
files using thegbif_dl.io
method. - a torch dataset class/pipeline to parse the dataset.