webdataset

webdataset copied to clipboard

It appears that `PIL` was not imported correctly in the `imageencoder`. Currently, it import `PIL` via `import PIL` directly and refer to `Image` via `PIL.Image` which cause the no attribute...

oeway

Why is `wds.Processor` not included in the `v2` or `main` branch.

2

I was going through the documentation and it points to using `wds.Processor` ([here](https://webdataset.github.io/webdataset/howitworks/)) to add a preprocessing pipeline to the data. However, in the `main` branch, this `Processor` class is...

abhayraw1

How to best subsample a dataset?

1

Hi, I'm trying to subsample a large dataset (LAION 400M) represented in the web dataset format and was hoping for some pointers. Just based on the documentation I am unsure...

ThaddaeusWiedemer

ddp_equalize

4

When using `DDP` during training, the documentation says to use the method `ddp_equalize` to make sure each worker process the same number of batches [[doc](https://webdataset.github.io/webdataset/multinode/#distributeddataparallel)]. However when running this: ```python...

cecabert

haooooooqi

Process hang with DDP training

4

I use webdataset in SpeechBrain. And I found there are 2 cases the process will hang in DDP training: 1. `num_workers` is set too large, set it to 1 is...

poor1017

Reading a wds with .npz files leads to pickle " Object arrays cannot be loaded when allow_pickle=False"

1

I am trying to implement a new dataloader with an s3 tar file being loaded. This tar file has .npz arrays and .cls with a class. .cls files load just...

charlescurt

Require support for the realse notes

1

I found that there are no release notes for each released version. It is hard for users/developers to know what happened between two different versions. Thanks a lot!

slyviacassell

multiple seeding / shuffle problems

2

I've been trying to debug and resolve a number of distributed training shuffle issues recently, I've found some alarming issues... 1. There is no way to have a reliable epoch...

rwightman

Processes not cleaned up?

16

Hi, I'm currently using Webdataset to stream data from S3 for my multi-gpu Pytorch training job (using multiprocessing Dataloader workers for each gpu like typical Pytorch jobs, etc.). For some...

thecooltechguy