img2dataset

img2dataset copied to clipboard

The downloading of LAION2B-multi and LAION1B-nolang stops at some point (no error)

3

Hi, when I download the LAION2B-multi dataset using Spark, it will stop at some point, but there is no error. Then I rerun the code by setting `incremental_mode="incremental"` and get...

viyjy

downloading large dataset with limited disk space

1

Hi @rom1504, for dataset like the Laion-400m or 5B, it is very hard to download them to a single disk when the disk space is limited. What I did was...

yao-eastside

type object 'SparkSession' has no attribute 'getActiveSession'

3

Hi, Thanks for your awesome work. I am using AWS EMR (with spark) to download the LAION5B dataset by following this [distributed mode](https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion5B.md). However, when I run the download.py in...

viyjy

rom1504

make it so shards can span several input files

1

rom1504

add redcaps example

3

https://redcaps.xyz/download

rom1504

Dataloader feature : make it possible to run training/inference in streaming

18

Useful to use img2dataset for inference directly without saving to disk

rom1504

consider building a generic sanity check tool

1

So people could run it on their output to check everything is as expected some things can be integrated directly as metrics in img2dataset, but maybe not everything

rom1504

provide more distributed strategies

2

for example * pure ssh * dask cluster ? * ray cluster ? follow up of https://github.com/rom1504/img2dataset/issues/20

rom1504

make tfrecord output format as fast as webdataset by using streaming and fsspec

1

current tfrecord format is 30% slower then webdataset when writing to a non local filesystem it also supports only a limited subset of file systems Let's figure out a way...

rom1504