webdataset

webdataset copied to clipboard

Fix typo in __init__.py and add version information

1

The package currently has a typo in `webdataset/__init__.py`. Also, it is not possible to query the package version using `webdataset.__version__`. The PR adds a proposal to retrieve the version when...

stes

How to write multi-process with ShardWriter?

3

980202006

I'd love to update our dep so we get [this commit](https://github.com/webdataset/webdataset/commit/7fadbfd13e7b506f1ecb33732a26423e0eab0e97), but it's not in any PyPI release. It looks like PyPI is a few versions behind.

reuben

resampled=True and with_length did not work for "exact epoch" when using shuffle

3

As explained the [document](https://github.com/webdataset/webdataset#multinode-training), I tried the resampled option for "exact epoch". This is the sample code. I executed this on single node. ``` import webdataset as wds dataset =...

kamo262

Resolves https://github.com/webdataset/webdataset/issues/179

mchalecki

gsutil cat intermittently fails

35

I keep getting the following error message randomly when using webdataset with gsutil cat. I have num_workers=4 in the dataloader, so it seems unlikely to be too many requests. Any...

egafni

Allow writing tiffs using TarWriter

1

I want to use tiff format because I have binary mask in float32 format which cannot be saved using png, jpg or ppm. The default encoder does not handle tiffs....

mchalecki

Webdataset fails on importing torch class

2

Webdataset does not list `torch` as dependency, so `torch` is not present when installing webdataset==0.2.5 with a non-Torch project. However, it then fails when doing ``` from webdataset import TarWriter...

f90

detshuffle epoch count the same across epochs

3

I'm using a pipeline as per #169, the core of it isn't disimilar to the OpenImages notebook (https://github.com/webdataset/webdataset/blob/ccfe88086cdb21a0dc23a6454ce3e3723b6b8033/notebooks/openimages.ipynb) except I have extra shuffle and worker/node splitters, etc. I was noticing...

rwightman

group_by_keys has a `None` handler arg, but it throws if there is a duplicate key, I'd rather continue of such an event happens (and it has in a dataset that's...

rwightman