[s3] LeoFS does not comply with S3 Spec of GET Bucket (List Object)

[windkit]

Description

LeoFS does not handle parameters Prefix and Delimiter correctly

Reference

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html

Details

In LeoFS, Prefix is assumed to be a "directory" path, Delimiter is assumed to be / while according to the spec, Prefix could an arbitrary string and Delimiter could be any character

Related Issues

https://github.com/leo-project/leofs/issues/367

[windkit]

access.log.txt

s3ql (https://bitbucket.org/nikratio/s3ql/) makes use of the bucket-get with prefix in startup, this issue is a blocker for using s3ql with LeoFS

[yosukehara]

I've confirmed embulk-input-s3 did not work due to this issue. It recursively called BUCKET-GET as below:

[BUCKET-GET]	test/	csv-	0	0	2018-04-24 18:15:43.374246 +0900	1524561343374244	200	2
[BUCKET-GET]	test/	csv-	0	0	2018-04-24 18:15:43.377736 +0900	1524561343377732	200	2
[BUCKET-GET]	test/	csv-	0	0	2018-04-24 18:15:43.384458 +0900	1524561343384455	200	3
[BUCKET-GET]	test/	csv-	0	0	2018-04-24 18:15:43.387213 +0900	1524561343387211	200	1
[BUCKET-GET]	test/	csv-	0	0	2018-04-24 18:15:43.394804 +0900	1524561343394801	200	2
[BUCKET-GET]	test/	csv-	0	0	2018-04-24 18:15:43.399243 +0900	1524561343399241	200	2
[BUCKET-GET]	test/	csv-	0	0	2018-04-24 18:15:43.403295 +0900	1524561343403293	200	1
[BUCKET-GET]	test/	csv-	0	0	2018-04-24 18:15:43.405492 +0900	1524561343405490	200	1
[BUCKET-GET]	test/	csv-	0	0	2018-04-24 18:15:43.408479 +0900	1524561343408476	200	2
[BUCKET-GET]	test/	csv-	0	0	2018-04-24 18:15:43.410316 +0900	1524561343410314	200	1
[BUCKET-GET]	test/	csv-	0	0	2018-04-24 18:15:43.412121 +0900	1524561343412120	200	1
[BUCKET-GET]	test/	csv-	0	0	2018-04-24 18:15:43.413881 +0900	1524561343413879	200	1
[BUCKET-GET]	test/	csv-	0	0	2018-04-24 18:15:43.417103 +0900	1524561343417102	200	1