webdataset

webdataset copied to clipboard

does webdataset have doc ?

3

does webdataset lib have doc or web to view api info?

habaohaba

This should be simple, but I can't figure it out. I need to use SFTTrainer with WebDataset. It works fine on a single node, but as soon as I use...

Ale9806

Unexpected Repetition and Missing Files in WebDataset with `resampled=True` and Shuffling Enabled

3

Hi, I hope you are doing well! I am encountering an issue when using WebDataset to load `.pt` files into an ML model. Below is a detailed description of the...

ManuelSokolov

My dataloader is not giving all the samples. Did I do some mistake defining the epoch length? I thought the error could be related with the resample, but in validation...

crscarpinteiro

Hello, thank you for developing this! I am new to webdatasets and I would like to know where I can find documentation on `webdataset` and `wids`. For now, the only...

javirk

Performance vs. TFRecord

2

We are benchmarking WebDataset vs. TFRecord on a tabular problem (the training set is a set of arrays) streaming from S3. We are seeing 2-3X degradation in performance between TFRecord...

guillaumeguy

How to prevent shuffling due to num_workers != 0 ? Can WebLoader objects be aggregated?

2

Hi, thank you for developing this! I have a directory with protein files where each protein contains shards, this should be the training data for a model. I am using...

ManuelSokolov

How Column Store work with multiple node training?

7

I tried the code in https://github.com/webdataset/webdataset/blob/main/examples/column-store.ipynb, but have trouble when i want to use Column Store in multi-node condition. The example said that any shuffling, decoding, etc. needs to happen...

krNeko9t

How to implement batch sampler on webdataset?

1

Hi everyone, hope you are doing well wanted to ask a technical question regarding webdataset. I was trying to implement a costum batch sampler function. The issue is the following,...

ManuelSokolov

lmdb_cached method & multiple dataloader workers

1

Hi, does the [`ldmb_cached`](https://github.com/webdataset/webdataset/blob/90346059ec6a64a950c37c252e38db64db00de0b/webdataset/compat.py#L294) method of the fluid interface work when using the [`WebDataset`](https://github.com/webdataset/webdataset/blob/90346059ec6a64a950c37c252e38db64db00de0b/webdataset/compat.py#L332) class and multiple `DataLoader` workers? In the code I see that once the entire dataset is...

pkrack