clyso
[BUG] Consistency check: false negatives on multipart uploads
Bug Report
Description
The consistency check occasionally reports false negatives for equal objects.
Analysis: The consistency check uses an objects ETag attribute to check wheather two objects are equal. Unfortunately, the ETag is not suitable for that purpose.
As described here (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity-upload.html#ChecksumTypes-Uploads) and here (https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.html), if an object was created via multipart upload or part copy, the ETag is not a checksum of the complete object, but constructed using the checksums of all individual parts. It can therefor be used to validate that an object has not changed since the ETag was last retrieved, or validate that two objects are equal, but not to validate that two objects are not equal.
Expected Behavior
The consistency check reports two objects with the same content as equal.
Actual Behavior
If an object was either uploaded as multipart upload or copied using multipart uploads, the consistency check might report the object as being different, even though they are equal. The actual behaviour depends on the clients used to upload the files, and the size and order of the upload parts.
Steps to Reproduce
- upload an object using multipart upload to two storages, with different part sizes
- run the chorus consistency check
- wait for the result. the consistency check will report the objects as not matching
- to verify the error, download both objects again and co0mpare them locally.
Environment
As of 2025-9-08, any current chorus version build from the main branch will do.
Additional Context
I am not aware of any mechanism to reliably copy an object in the same way as it was uploaded. Nor am I aware of any way to reliably check two objects equality based on checksums, but I might be missing something.
One way to deal with this might be to just ignore the ETag and only rely on the object size if the ETag is a composite checksum (thats what rclone does).
An other way would be to download the file and actually download the validate both both objects, but that should be optional since it could use a lot of resources.
I can think of several ways to mitigate the issue by improving the copy process and using either a local state or custom object metadata. But none of them is be 100% reliable and will not help for objects which have not been copied by the chorus proxy.
The consistency checker is having behavior similar to rclone's check. Probably the only difference we are having now is that rclone in addition does the HEAD request and looks for md5 checksum metadata header, which should be set by user. We can improve the solution of course.
The actual behaviour depends on the clients used to upload the files, and the size and order of the upload parts.
Iirc the order shouldn't be the issue, but different client/server configuration for multipart upload can be the case, at it will affect the composite result.
One way to deal with this might be to just ignore the ETag and only rely on the object size if the ETag
This is something we can probably do. I can suggest to introduce couple of flags. Right now it would be easy to add capability to configure check and do gradually
- list of objects matching
- size matching
- etag matching depending on needs and conditions.
Later on we can support
- checksum headers
- metadata fields as well, which could be set by user or, in case of AWS S3, by checksum computation batch job.
https://github.com/clyso/chorus/pull/134 covers optional disabling of etag and size checks.