skypilot icon indicating copy to clipboard operation
skypilot copied to clipboard

[Storage] file mount with object store download the data again by `sky launch` twice

Open Michaelvll opened this issue 1 year ago • 0 comments

sky launch twice the following yaml will download the data again. This will cause unnecessary costs for downloading. Maybe we should consider using aws s3 sync instead of aws s3 cp to do it incrementally.

resources:
  cloud: AWS
  accelerators: V100:8
  
file_mounts:
  /data/imagenet.tar.gz: s3://sky-imagenet-tar/sky-imagenet.tar.gz
  
setup: |
  cd /data
  tar xvzf /data/imagenet.tar
  cd -
  git clone https://github.com/pytorch/examples
  

Michaelvll avatar Jul 19 '22 04:07 Michaelvll