skypilot
skypilot copied to clipboard
[Storage] file mount with object store download the data again by `sky launch` twice
sky launch
twice the following yaml will download the data again. This will cause unnecessary costs for downloading. Maybe we should consider using aws s3 sync
instead of aws s3 cp
to do it incrementally.
resources:
cloud: AWS
accelerators: V100:8
file_mounts:
/data/imagenet.tar.gz: s3://sky-imagenet-tar/sky-imagenet.tar.gz
setup: |
cd /data
tar xvzf /data/imagenet.tar
cd -
git clone https://github.com/pytorch/examples