webdataset

webdataset copied to clipboard

Collate function confusion

3

Hello, im trying to figure out how the [default_collation_fn](https://webdataset.github.io/webdataset/api/webdataset/iterators.html) works: The first line of comment says ` """Take a collection of samples (dictionaries) and create a batch.` But the first...

KayARS

ShardWriter works only with local paths

2

As far as I see, ShardWriter cannot write to remote urls, although inside itself it uses TarWriter. May be there is some logical reason about this, but it would be...

artemkurylev

The exception handler is currently away from the `tar_file_iterator`. I am not sure that it is intentional or just a typo. I feel that it is a typo since the...

airsplay

Hello, Thanks a lot for such a wonderful and useful library. I'm pretty new to this, and the library I'm using (video2dataset) uses webdataset as the base. I'm sorry for...

rohit901

According to the [thread](https://github.com/webdataset/webdataset/issues/250), i'm work in the `resample=True` + `.with_epochs(n)` method. However, how to properly setup n in `.with_epochs(n)` remains unsolved in the above thread. My first question :...

HuangChiEn

Iterable is used in the is_iterable function, but the import is missing

alexanderwerning

wids cannot open SAM dataset

7

Hi author, Thanks for designing such a flexible and efficient dataloader. WIDS used to work fine in my training pipelines, however, I recently notice a werid error when loading [SAM...

Lyken17

Sharded Dataset Has Long Delay Before First Batch (and caching=False)

8

I have a sharded dataset for 1000 large images (675 MB each). The shards were capped at 3 GB, and each shard contains 7 images (143 shards). Using WebDataset with...

cupdike

Arg `repeat` of class `WebDataset` is not used

3

The `repeat` is ignored in `__init__` function https://github.com/webdataset/webdataset/blob/039d74319ae55e5696dcef89829be9671802cf70/webdataset/compat.py#L102

ychfan

how to use wids on local dataset

3

webdataset==0.2.86 ``` shards = [dict(url="lines-000000.tar", nsamples=250)] ds = wids.ShardListDataset(shards, keep=True) print(ds[0]) ```` then happen ``` File "/usr/local/lib/python3.6/dist-packages/wids/wids.py", line 515, in __getitem__ shard, inner_idx, desc = self.get_shard(index) File "/usr/local/lib/python3.6/dist-packages/wids/wids.py", line 510,...

WudiJoey