S3 Multipart Upload Fails When ContentLength is Provided

[mritterfigma]

Checkboxes for prior research

• [X] I've gone through Developer Guide and API reference
• [X] I've checked AWS Forums and StackOverflow.
• [X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

S3 Multipart Uploads are not working in the V3 SDK. They are not working because completing the multi-part upload fails due to malformed XML:

MalformedXML: The XML you provided was not well-formed or did not validate against our published schema
at throwDefaultError (/src/upnode/node_modules/@aws-sdk/client-s3/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)
at deserializeAws_restXmlCompleteMultipartUploadCommandError (/src/upnode/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:3087:43)

This is an SDK bug that has been described before (https://github.com/aws/aws-sdk-js-v3/issues/1814, https://github.com/aws/aws-sdk-js-v3/issues/2445) and was allegedly fixed by https://github.com/aws/aws-sdk-js-v3/pull/2835. Unfortunately, I am finding that it still persists on v3.165.0, which contains the commit (https://github.com/aws/aws-sdk-js-v3/commit/7e634cfab78d0a82864f828c4c8c10c9d0cccc97).

Is there more action I can take to get this to work? It is blocking adoption of the V3 SDK for my team.

SDK version number

@aws-sdk/[email protected]

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v16.15.1

Reproduction Steps

To reproduce, try to use lib-storage's Upload with a file greater than 5MB. If the file is too small, it will not trigger multi-part upload.

import { S3Client } from '@aws-sdk/client-s3'
import { Upload } from '@aws-sdk/lib-storage'

upload = new Upload({
  client: new S3Client({}),
  params: const params = {
    Bucket: "xxxx",
    Key: "yyyy",
    Body: "zzzz",
  }
})
await upload.done()

Observed Behavior

The upload hangs for a long time, then finally fails with the following error:

MalformedXML: The XML you provided was not well-formed or did not validate against our published schema
at throwDefaultError (/src/upnode/node_modules/@aws-sdk/client-s3/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)
at deserializeAws_restXmlCompleteMultipartUploadCommandError (/src/upnode/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:3087:43)

Expected Behavior

Expected that the multipart upload succeeds.

Possible Solution

It is not clear why the fix PR is not working. Perhaps it depends on another npm module that must be updated in order to get the desired behavior?

Additional Information/Context

No response

[ajredniwja]

Hey @mritterfigma thanks for opening this issue, I was not able to unable to reproduce this issue, running the code like:

import { S3Client } from '@aws-sdk/client-s3';
import { Upload } from '@aws-sdk/lib-storage';

(async() => {
    const s3client = new S3Client({
        region: 'us-west-2',
    });


    const Key = `${Date.now()}-new-key`;
    let upload = new Upload({
        client: s3client,
        params: {
            Key: Key,
            Bucket: 'bucket',
            Body: 19mb
        }
    });

    upload.on("httpUploadProgress", (progress) => {
        console.log(progress);
    });

    const uploadResult = await upload.done();

    // debugger;
    console.log("done uploadingg", uploadResult);
})();

Response:

{
  loaded: 5242880,
  total: 20164608,
  part: 1,
  Key: '173u3uu3i-new-key',
  Bucket: 'bucket'
}
{
  loaded: 9678848,
  total: 20164608,
  part: 4,
  Key: '173u3uu3i-new-key',
  Bucket: 'bucket'
}
{
  loaded: 14921728,
  total: 20164608,
  part: 3,
  Key: '173u3uu3i-new-key',
  Bucket: 'bucket'
}
{
  loaded: 20164608,
  total: 20164608,
  part: 2,
  Key: '173u3uu3i-new-key',
  Bucket: 'bucket'
}
done uploadingg {
  '$metadata': {
    httpStatusCode: 200,
    requestId: undefined,
    extendedRequestId: 'KezdjkhdoqkjahdHwfakljaskld=',
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Bucket: 'aj-bar',
  BucketKeyEnabled: undefined,
  ChecksumCRC32: undefined,
  ChecksumCRC32C: undefined,
  ChecksumSHA1: undefined,
  ChecksumSHA256: undefined,
  ETag: '""',
  Expiration: 'expiry-date="Tue, 20 Sep 2022 00:00:00 GMT", rule-id=""',
  Key: '16476787878787-new-key',
  Location: 'https://bucket.s3.us-west-2.amazonaws.com/4823232-new-key',
  RequestCharged: undefined,
  SSEKMSKeyId: undefined,
  ServerSideEncryption: undefined,
  VersionId: 'kfsdkjfnkdnf_89342809'
}

[mritterfigma]

Thanks @ajredniwja for taking a look. I have confirmed that this only happens when ContentLength is specified in the params. Could you please try again with specification of the ContentLength? I was able to confirm that the ContentLength I am passing matches the file size as reported by my Macbook.

Perhaps I can just leave off ContentLength, since I don't think I need it with lib-storage.

[JacopoBonta]

@mritterfigma can be related to this one?

[AJB99]

@ajredniwja Did you even read the original post? It clearly states that this bug only occurs when ContentLength is provided as a param to the Upload command.

I ran into this issue myself and just wasted two days trying to figure out what the issue was. Perhaps AWS might want to at least document this "feature". Or possibly grace us with a Dev Support Engineer that isn't so lazy that they don't even bother to read the issues before providing a "solution".

I can confirm @mritterfigma findings. When you remove the ContentLength param from the new Upload constructor then multipart uploads work as expected.

Can anyone from AWS please comment why this is, and what side effects removing the ContentLength param could trigger.

[andyslack]

Yes, I have the same issue, Upload works fine for smaller files (e.g. 1-2mb) but when I try to upload a 50mb file, I get the same The XML you provided was not well-formed or did not validate against our published schema error.

I am not passing ContentLength but I am still getting the error.

[DougReeder]

FWIW, I am also seeing this — if I use Upload for a stream longer than about 5.1MB, the stream will have all its bytes read, then Upload appears to hang for a number of minutes, then I get the unhelpful error message about XML. Not sending the Content-Length appears to allow it to work.

It's disappointing that this hasn't been fixed after a year and a half — I'm looking at chucking your SDK in favor of calling the REST endpoints directly.

[usman-nst]

"I am facing the same issue"