cloudpathlib
cloudpathlib copied to clipboard
Recursion error on "fake" s3 directories
This was brought up by @remi-braun in #148.
Sometimes, listing a directory with the S3Client
results in a recursion error. This was seen on Ceph where "faked" S3 directories are more common.
@GeorgeSabu found that this is likely the same root cause as what was fixed in #190. We assume that all the "content" items from the API are files, but they can be directories as well. The way to detect which is a directory is to see if the size is 0, as is implemented here: https://github.com/ElucidataInc/cloudpathlib/commit/9c11c6af2b0ac713ddcd950123d1db3b17515efa
@pjbull I am interested in contributing, here is the PR.
@pjbull per my comment here, I think that the root cause for this issue may be misidentified. I think it might actually be the issue with download_to
+ iterdir
that was identified in #204, and in that case it would be fixed by #202.
Between #202 and #302, I think this should be fixed now. Closing unless we see this again.