aws-s3-proxy
aws-s3-proxy copied to clipboard
GET_ALL_PAGES_IN_DIR doesn't work for "dirs" full of "subdirs"
The fix in #31 doesn't seem to work if you attempt to list a directory (prefix) full of only subdirectories (longer prefixes) as page.Contents
is 0 and it stops after the first page.
Various AWS SDK documentation and examples (e.g. here on GitHub just pass in a function that always returns true
, apparently the SDK will stop calling it after the last page.
A better fix is something like this:
--- a/internal/service/amazon-s3.go
+++ b/internal/service/amazon-s3.go
@@ -37,7 +37,7 @@ func (c client) S3listObjects(bucket, prefix string) (*s3.ListObjectsOutput, err
func(page *s3.ListObjectsOutput, lastPage bool) bool {
result.CommonPrefixes = append(result.CommonPrefixes, page.CommonPrefixes...)
result.Contents = append(result.Contents, page.Contents...)
- return len(page.Contents) == 1000
+ return true
})
return result, err
}