dataloader-benchmarks
dataloader-benchmarks copied to clipboard
DL Dataloader Benchmarks
dataloader-benchmarks
This is a benchmarking suite for data loading libraries.
Set up
Environmental variables
Create a .env
file with the following information
DOCKER_NAME=<name of org>/<name of container>:<version>
DYNACONF_AWS_ACCESS_KEY_ID=<aws id>
DYNACONF_AWS_SECRET_ACCESS_KEY=<aws secret>
DYNACONF_BUCKET_NAME=<bucket name in aws> # needs to exist before running experiments
Running locally
- Clone this repository
- Build the docker container:
./scripts/build.sh
- Run the container:
./scripts/run.sh
- Run all the experiments:
./experiments/run_all.sh
Running on AWS
- Create the file
~/.aws/credentials
with the following content:
[default]
aws_access_key_id = <aws id>
aws_secret_access_key = <aws secret>
- Make sure that an S3 bucket is created with the name defined above and that it is accessible with the credentials provided.
- Download the
get_ecr
script to fetch the latest docker image:wget https://raw.githubusercontent.com/kiedanski/dataloader-benchmarks/main/scripts/get_erc.sh && chmod +x get_ecr.sh
- Download the latest docker image locally:
./get_ecr.sh
- Download the run script:
wget https://raw.githubusercontent.com/kiedanski/dataloader-benchmarks/main/scripts/run.sh && chmod +x run.sh
- Execute the run command to get into the docker container:
./run.sh
- Run all the experiments:
./experiments/run_all.sh
Collecting results and plotting
Inside the container run:
-
python src/plots/download_results.py
-
python src/plots/generate_plots.py
Implemented Libraries and Datasets
Pytorch | FFCV | Hub | Deep Lake | Torchdata | Webdataset | Squirrel | NVIDIA DALI | ||
---|---|---|---|---|---|---|---|---|---|
CIFAR-10 | default | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
remote | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ❓ | ❌ | |
filtering | ✅ | ❓ | ✅ | ✅ | ✅ | ✅ | ❓ | ❌ | |
multi-gpu | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | |
RANDOM | default | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
remote | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ❓ | ❌ | |
filtering | ✅ | ❓ | ✅ | ✅ | ✅ | ✅ | ❓ | ❌ | |
multi-gpu | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | |
CoCo | default | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
remote | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❓ | ❌ | |
filtering | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ❓ | ❌ | |
multi-gpu | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ |