aws-s3-proxy icon indicating copy to clipboard operation
aws-s3-proxy copied to clipboard

GET_ALL_PAGES_IN_DIR doesn't work for "dirs" full of "subdirs"

Open hartzell opened this issue 1 year ago • 0 comments

The fix in #31 doesn't seem to work if you attempt to list a directory (prefix) full of only subdirectories (longer prefixes) as page.Contents is 0 and it stops after the first page.

Various AWS SDK documentation and examples (e.g. here on GitHub just pass in a function that always returns true, apparently the SDK will stop calling it after the last page.

A better fix is something like this:

--- a/internal/service/amazon-s3.go
+++ b/internal/service/amazon-s3.go
@@ -37,7 +37,7 @@ func (c client) S3listObjects(bucket, prefix string) (*s3.ListObjectsOutput, err
                func(page *s3.ListObjectsOutput, lastPage bool) bool {
                        result.CommonPrefixes = append(result.CommonPrefixes, page.CommonPrefixes...)
                        result.Contents = append(result.Contents, page.Contents...)
-                       return len(page.Contents) == 1000
+                       return true
                })
        return result, err
 }

hartzell avatar Apr 04 '23 14:04 hartzell