skypilot
skypilot copied to clipboard
[Storage] Fix public bucket source check in SkyPilot Storage
Closes #977.
GetPublicAccessBlock
is no longer a reliable method to check if a s3 bucket is public. Buckets can be publicly readable yet not allow access to GetPublicAccessBlock
. I'm not sure what caused it (and I can't find a clear reason).
Some buckets still allow it (e.g., s3://tcga-2-open
, which is used in our smoke tests and thus was passing), but others have blocked it (e.g., s3://digitalcorpora
, s3://imagenet-bucket
).
Using head_bucket
API now, as recommended by boto3 docs.
Also added storage tests for non-existent and private buckets.
Tested:
- [x]
pytest test_smoke.py::TestStorageWithCredentials
- [x] YAML in #977
- [ ] All smoke tests