cloudpathlib icon indicating copy to clipboard operation
cloudpathlib copied to clipboard

LocalS3Path doesn’t support glob on non-existing directories

Open Quickblink opened this issue 1 year ago • 1 comments

The implementations of S3Path and LocalS3Path diverge in the following way which disqualifies using LocalS3Path as a testing tool.

from cloudpathlib.s3 import S3Path

path = S3Path("s3://this/path/does/not/exist")
path.mkdir(parents=True) # does nothing (because S3 doesn't have directories)
print(list(path.glob("*"))) # returns []
from cloudpathlib.local import LocalS3Path

path = LocalS3Path("s3://this/path/does/not/exist")
path.mkdir(parents=True) # again, does nothing
print(list(path.glob("*"))) # raises exception

To fix this in my own codebase, I added a check to LocalS3Path.glob to return an empty generator when the path is not a directory. However, I’m unsure if this fixes the issue in all cases and there might be other issues caused by directories not existing in S3 but locally.

Quickblink avatar Mar 02 '24 11:03 Quickblink

Verified this bug.

I'd be happy with a fix that overrides glob in the LocalPath implementation to try globbing and return an empty generator on FileNotFoundError.

pjbull avatar Mar 02 '24 19:03 pjbull