cloudpathlib icon indicating copy to clipboard operation
cloudpathlib copied to clipboard

Recursion error on "fake" s3 directories

Open pjbull opened this issue 3 years ago • 2 comments

This was brought up by @remi-braun in #148.

Sometimes, listing a directory with the S3Client results in a recursion error. This was seen on Ceph where "faked" S3 directories are more common.

@GeorgeSabu found that this is likely the same root cause as what was fixed in #190. We assume that all the "content" items from the API are files, but they can be directories as well. The way to detect which is a directory is to see if the size is 0, as is implemented here: https://github.com/ElucidataInc/cloudpathlib/commit/9c11c6af2b0ac713ddcd950123d1db3b17515efa

pjbull avatar Feb 02 '22 20:02 pjbull

@pjbull I am interested in contributing, here is the PR.

GeorgeSabu avatar Feb 03 '22 04:02 GeorgeSabu

@pjbull per my comment here, I think that the root cause for this issue may be misidentified. I think it might actually be the issue with download_to + iterdir that was identified in #204, and in that case it would be fixed by #202.

jayqi avatar Feb 06 '22 16:02 jayqi

Between #202 and #302, I think this should be fixed now. Closing unless we see this again.

pjbull avatar Dec 18 '22 23:12 pjbull