s3proxy icon indicating copy to clipboard operation
s3proxy copied to clipboard

Published 20 hours ago • verified publisher icon gaul

Restore OPAQUE_MARKERS handling for list objects V1

Open larshagencognite opened this issue 1 year ago • 7 comments

We need this to support V1 clients that use the last object name in the response as a marker for the next page (which is allowed according to the S3 docs).

larshagencognite avatar Oct 24 '23 09:10 larshagencognite

@Decard6 This partially reverts changes in https://github.com/gaul/s3proxy/pull/531

larshagencognite avatar Oct 24 '23 09:10 larshagencognite

I'm concerned that s3-tests isn't exercising this behavior. Can you look at these and see what's missing?

gaul avatar Oct 24 '23 11:10 gaul

Do we have automated testing against a real azure blob store?

larshagencognite avatar Oct 25 '23 07:10 larshagencognite

@gaul Could you give me some pointers to where this should be tested?

larshagencognite avatar Oct 25 '23 08:10 larshagencognite

The ideal way to test this is https://github.com/ceph/s3-tests which I periodically rebase onto. But I don't have any automated integration tests against Azure presently.

Alternatively maybe we could parameterize some S3Proxy listObjects tests to toggle OPAQUE_MARKERS? I'm happy to merge this PR but just want to make sure that we have some way to avoid future regressions.

gaul avatar Oct 25 '23 13:10 gaul

@larshagencognite could you rebase this change?

gaul avatar Jan 12 '24 03:01 gaul

@Decard6 any thoughts here?

gaul avatar Mar 10 '24 07:03 gaul