skypilot icon indicating copy to clipboard operation
skypilot copied to clipboard

[Storage] Fix public bucket source check in SkyPilot Storage

Open romilbhardwaj opened this issue 1 year ago • 0 comments

Closes #977.

GetPublicAccessBlock is no longer a reliable method to check if a s3 bucket is public. Buckets can be publicly readable yet not allow access to GetPublicAccessBlock. I'm not sure what caused it (and I can't find a clear reason).

Some buckets still allow it (e.g., s3://tcga-2-open, which is used in our smoke tests and thus was passing), but others have blocked it (e.g., s3://digitalcorpora, s3://imagenet-bucket).

Using head_bucket API now, as recommended by boto3 docs.

Also added storage tests for non-existent and private buckets.

Tested:

  • [x] pytest test_smoke.py::TestStorageWithCredentials
  • [x] YAML in #977
  • [ ] All smoke tests

romilbhardwaj avatar Aug 17 '22 06:08 romilbhardwaj