Sky Storage Download Feature

[michaelzhiluo]

Add a Sky Storage Download Feature. Also fixes #932.

To download a folder from a bucket using Sky Storage, run:

from sky.data import storage, StoreType
storage_obj = storage.Storage(name='imagenet-bucket')
storage_obj.add_store(StoreType.S3)
storage_obj.download('~/Downloads/hello')

(also works for GCS)

Tests

• [x] Code above for both S3 and GCS

[romilbhardwaj]

I think we should have a design review for the storage download interface before we proceed with implementing this feature. I believe the interface should allow the specification of both, the remote path to download and the local path where to place the download.

I have detailed this design in a document here, please feel free to leave comments and add new proposals.