litestream icon indicating copy to clipboard operation
litestream copied to clipboard

Update S3 multipart upload test once moto issue #8762 is resolved

Open corylanou opened this issue 5 months ago • 0 comments

Context

We currently work around a moto bug in our S3 integration tests by limiting test file sizes to 4MB to avoid triggering multipart uploads.

The Problem

Moto issue #8762 causes multipart upload tests to fail with AWS SDK v2 due to incorrect checksum handling:

  • Composite checksums for multipart uploads should have a -X suffix to distinguish them from full object checksums
  • Moto doesn't add this suffix, causing AWS SDK v2 checksum validation to fail with errors like:
    checksum did not match: algorithm CRC32, expect PnvBcQ==, actual KxHXkQ==
    

Current Workaround

In replica_client_test.go, the TestReplicaClient_S3_UploaderConfig test uses a 4MB file instead of a larger file to avoid triggering multipart uploads:

// Create a 4MB test file. We use 4MB instead of a larger file to avoid
// triggering multipart uploads which have a known issue with moto mock server.
// Moto issue #8762: composite checksums don't have the -X suffix, causing
// AWS SDK v2 checksum validation to fail.
// Reference: https://github.com/getmoto/moto/issues/8762

Action Required Once Moto is Fixed

Once moto issue #8762 is resolved, we should:

  1. Update the test to use a larger file size (e.g., 10MB or more) to properly test multipart upload functionality:

    // Change from:
    size := 4 * 1024 * 1024  // 4MB
    
    // To:
    size := 10 * 1024 * 1024 // 10MB (or larger)
    
  2. Update or remove the workaround comment to reflect that the moto issue has been fixed

  3. Verify the test passes with multipart uploads:

    ./etc/s3_mock.py go test -v ./replica_client_test.go -integration s3 -run TestReplicaClient_S3_UploaderConfig
    
  4. Consider adding additional multipart upload tests to ensure our S3 client properly handles large files with custom PartSize and Concurrency settings

Benefits of Fixing This

  • Better test coverage for multipart upload scenarios
  • Validation that our custom PartSize and Concurrency settings work correctly with large files
  • More realistic testing of production scenarios where large files are uploaded

Tracking

  • Moto issue: https://github.com/getmoto/moto/issues/8762
  • Moto PR (draft): https://github.com/getmoto/moto/pull/8875

corylanou avatar Aug 13 '25 20:08 corylanou