NSFS | S3 | Versioning: CopyObject after multipart upload of a versioned object leads to invalid data content

[hseipp]

Environment info

• NooBaa Version: 5.15.4
• Platform: CentOS 8, deployed using https://github.com/IBM/StorageScaleVagrant

Actual behavior

Content of the copied object whose source was uploaded to a version-enabled bucket using multi-part upload is not identical to the source object:

E       AssertionError: assert 'bIMcUvjvnUme...cjoUdmaMHmNZD' == 'avLZIzDyrPsQ...cjoUdmaMHmNZD'
E         
E         - avLZIzDyrPsQTmrmiOZHrVdSZCtRFVsosTSLGjluRPBZGbXZNTKAKSWyhuegjyevNmYTqVxjrYlMllwCdCjefarPnMGBgGnnQebJjAdaIVSqUbhSkwakohDylIGMTJKgpjxJlmHEzycMbQdOvGdimZxcOFJnkBYBSvQEXYifdmBKfrgJOZXdVcvzbjsZFBOxvOQNqsAKqfUYXUBEBWzfQdGMgFBnXfiJWUrWOwLWsBaAPIAmkjzFVtHXETyLiLgZkKvjtbBkfDcFLxGBgnIeMjQBJYNzCVkzidUdaPqzaVYawKkIbRKqZsyjPSrpXrqIIECWMzhPQNkjyuEHfCSFeJPIqfBfxpPBooDkhsmDzmIMllnrYugjdonucmiOXipomGLWIshDfqhTWCZbjBXNusYadCgZWxyBpyloWHbDHGBGKFzqiFTvRemtGPKJSYwyDEFhjiJPTVeBGfbIaUcHoMDoIpSPFSkGgmOiNhHAluJAvymOtuOhvpobXaUbGoQCgLLAXBJvpBumNZJNBGsEyIoqnSxzoQlgqnZxGwZMvsGxoKkIOXlzLoAsscwzDB...

Expected behavior

Content of the copied object should be identical to the source object, assertion should pass.

Steps to reproduce

Run Ceph s3-tests test_object_copy_versioning_multipart_upload():

S3TEST_CONF=s3tests.conf tox -- s3tests_boto3/functional/test_s3.py::test_object_copy_versioning_multipart_upload

More information - Screenshots / Logs / Other output

See "all" logging level noobaa log attached - look at log entries at 15:34:00 and later:

noobaa_20240828_1748.log.gz