aws-sdk/lib-storage does not upload full item

[DahShahar]

Describe the bug

Occasionally (<.1% of the time), the Upload class uploads only part of the object. Suspiciously, it only uploads 5mb of an otherwise larger object. I have had this occur where I expect the object to be 5.4mb, 8.1mb, and 10.2mb.

Our code is simple. We get a Javascript object, JSON.stringify() it, and upload using SSE-KMS.

When this occurs, there is no stack trace or indication of failure by the library at all.

Expected Behavior

The entire object should get uploaded, or an error thrown if something goes wrong so that I know what happened.

Current Behavior

The application uploads less than the full object, silently discarding the rest of the object. There are no logs indicating failure.

Reproduction Steps

This does not occur often, so I don't have a great reproducible set-up. Here is basically what my code is doing:

  // this lives within a class
class Uploader {

  constructor(private readonly bucket) { }
  
  public async upload(
    name: string,
    input: Record<string, any>,
    kmsArn: string,
  ): Promise<void> {
    const body = JSON.stringify(input);
 
    const input: PutObjectCommandInput = {
      Bucket: this.bucket,
      Key: name,
      Body: body,
      SSEKMSKeyId: kmsArn,
      ServerSideEncryption: 'aws:kms',
    };
 
    const start = Date.now();
 
    try {
      await new storage.Upload({
        client: this.s3,
        params: input,
      }).done();
    } catch (err) {
      console.error({ err }, 'There was an error uploading');
      throw err;
    } finally {
      console.log(`Took ${Date.now() - start} seconds to complete`);
    }
  }
}

Possible Solution

No response

Additional Information/Context

This code is running on AWS Lambda, if that changes anything.

SDK version used

3.88.0

Environment details (OS name and version, etc.)

AWS Lambda, Nodev14

[yenfryherrerafeliz]

Hi @DahShahar, thanks for opening this issue. I tried multiple times to reproduce the reported behavior, but in my tests I got successful results each try. I know you mentioned this issue just happen sometimes so that I will keep my eyes on this issue to see if more people from the community react or report this.

Here is the code I used to reproduce:

import {S3Client} from "@aws-sdk/client-s3";
import  {Upload} from "@aws-sdk/lib-storage";

const bucket = 'BUCKET';
const input = {
    Bucket: bucket,
    Key: 'SAMPLE-KEY',
    Body: '#'.repeat((1024 * 1024) * 5.4),
    SSEKMSKeyId: '',
    ServerSideEncryption: 'aws:kms'
};
const start = Date.now();
const s3Client = new S3Client({
    region: 'REGION'
});
const upload = new Upload({
    client: s3Client,
    params: input
});
const response = await upload.done();

console.log('Response: ', response);

Thanks!

[kontrollanten]

For the record: I've experienced a quite similar issue, reported in https://github.com/Chocobozzz/PeerTube/issues/5227

I had configured partSize to 2 gb though. The file was uploaded without any error, but the uploaded file was only 408,9 mb instead of 2,4 gb. I tried in three times with the same file and the same issue appeared. When I changed partSize to 500 mb the issue was solved. This happened in version 3.72.0, so maybe it's not related to this issue and solved in the latest version.

[yenfryherrerafeliz]

Hi @DahShahar, I have not seen many people reacting to this report, but would it be possible for you to please confirm if this issue is still happening for you?

I look forward to your response.

Thanks!

[DahShahar]

Hi, my team switched away from using this upload to just using a regular PutObject call. I'd be interested in reverting to this if it were fixed, but I don't know what the issue was or if there was any changes that might have fixed this.

I'm hesitant to migrate back, but we can do it if you think this should be ready now.

[yenfryherrerafeliz]

Hi @DahShahar, I am still not able to reproduce this issue, but if you know a specific scenario that could reproduce it please share it with me so I can better help you. I would also recommend to upgrade the latest SDK version in your local and bundle it along with your deployment, since the SDK version embedded in lambda is not up to date. The version in lambda is still in v3.188.0.

Thanks!

[github-actions[bot]]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.