Inconsistent After Completing Multipart Upload

[dw214]

Describe the bug

inconsistent object availability after completing multipart upload with previously abandoned uploadId

Regression Issue

• [ ] Select this option if this issue appears to be a regression.

Expected Behavior

After step 4, complete_multipart_upload for uploadId2 should succeed, and get_object (step 5) should return the object data uploaded via uploadId2.

Current Behavior

complete_multipart_upload for uploadId2 succeeds (HTTP 200), but get_object returns NoSuchKey error. The object f.txt remains inaccessible despite the successful completion of uploadId2

Reproduction Steps

1. Call create_multipart_upload twice for the same object key (e.g., d1/d2/f.txt), generating two distinct uploadIds (uploadId1 and uploadId2).
2. Upload parts and complete uploadId1 using complete_multipart_upload.
3. Delete the object f.txt (via delete_object).
4. Attempt to complete uploadId2 using complete_multipart_upload (no parts uploaded for uploadId2).
5. Call get_object for f.txt.

Possible Solution

S3 may invalidate abandoned uploadIds when the object is deleted, but complete_multipart_upload does not propagate this invalidation, leading to a mismatch between the API response and the actual object state.

Additional Information/Context

No response

SDK version used

1.28.63

Environment details (OS name and version, etc.)

Rocky 8

[github-actions[bot]]

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.

[adev-code]

Hi @dw214, thanks for reaching out. I have replicated the same issue and got the same result that I was not able to get the object. Please take note that as per S3, the object multipart upload must be complete https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html . Please let me know if you have any other questions. Thanks.

[dw214]

Hi @dw214, thanks for reaching out. I have replicated the same issue and got the same result that I was not able to get the object. Please take note that as per S3, the object multipart upload must be complete https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html . Please let me know if you have any other questions. Thanks.

Thanks for your answer! I confirmed that complete_multipart_upload() was called after the multipart upload was complete, but the object still could not be get_object(). This suggests that the problem may be related to the interaction logic of boto3 rather than a missing step in the process.

Thanks.

[adev-code]

Thanks for the reply. I have reached out to the S3 Service Team since the response and behavior is actually coming from S3 itself. An update will be given once the S3 Team gives an update.

Internal Ref: P304176980

[adev-code]

Thanks for the patience. The S3 Service Team has mentioned that this is an expected behavior to maintain the eventual consistency, as explained on the doc. As mentioned from their doc:

• Another example of when a concurrent multipart upload request can take precedence is if another operation deletes a key after you initiate a multipart upload with that key. Before you complete the operation, the complete multipart upload response might indicate a successful object creation without you ever seeing the object.

The object would not be visible and hence get_object fails.

[github-actions[bot]]

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.