`PutObjectCommand` fails when passing a `ReadableStream<Uint8Array>` with a TypeError only at runtime

[Sophon96]

Checkboxes for prior research

• [X] I've gone through Developer Guide and API reference
• [X] I've checked AWS Forums and StackOverflow.
• [X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

Attempting to pass Response.body to the Body parameter of the PutObjectCommand constructor results in a TypeError at runtime, but not at compile time:

TypeError [ERR_INVALID_ARG_TYPE]: The first argument must be of type string or an instance of Buffer, ArrayBuffer, or Array or an Array-like Object. Received an instance of ReadableStream

SDK version number

@aws-sdk/[email protected]

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v18.12.1

Reproduction Steps

Here's a minimal reproduction repo: https://github.com/Sophon96/s3fetchandputrepro

Observed Behavior

There are no errors at TS compile time, but the following error is produced at runtime:

TypeError [ERR_INVALID_ARG_TYPE]: The first argument must be of type string or an instance of Buffer, ArrayBuffer, or Array or an Array-like Object. Received an instance of ReadableStream
    at __node_internal_captureLargerStackTrace (node:internal/errors:484:5)
    at new NodeError (node:internal/errors:393:5)
    at Function.from (node:buffer:328:9)
    at writeBody (C:\Users\User2\Projects\s3_types_repro\node_modules\.pnpm\@[email protected]\node_modules\@smithy\node-http-handler\dist-cjs\index.js:147:28)
    at writeRequestBody (C:\Users\User2\Projects\s3_types_repro\node_modules\.pnpm\@[email protected]\node_modules\@smithy\node-http-handler\dist-cjs\index.js:128:5)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  code: 'ERR_INVALID_ARG_TYPE'
}

Expected Behavior

PutObjectCommand should accept the ReadableStream<Uint8Array> of Response.body without error.

Possible Solution

No response

Additional Information/Context

No response

[aBurmeseDev]

Hi @Sophon96 - thanks for reaching out.

The issue you're facing is caused by a type mismatch between the expected input type for the Body parameter of the PutObjectCommand constructor and the actual type of the Response.body you're trying to pass.

The Body parameter of the PutObjectCommand constructor expects one of these types: string | Uint8Array | Buffer | Readable

The TypeScript compiler doesn't catch this error because the Body parameter is defined as a union type that includes any (or unknown in newer TypeScript versions). This means that any type of value can be assigned to Body without causing a compile-time error.

---

Here's how you can convert the ReadableStream before passing it to PutObject in your current code:

import { Readable } from 'stream';

// ...

async function fetchAndPut() {
  const response = await fetch("https://picsum.photos/1600/900.webp");

  // ...

  const stream = response.body as Readable;
  const uploadParams: PutObjectCommandInput = {
    Bucket: "fetchandputrepro",
    Key: "image1.webp",
    Body: await streamToBuffer(stream), // Convert the stream to a Buffer
    ContentType: "image/webp",
    ContentLength: contentLength,
  };

  await s3Client.send(new PutObjectCommand(uploadParams)).then(
    (data) => console.log(`Etag: ${data.ETag}`),
    (err) => console.error(`response.body error: ${err}`)
  );
}

async function streamToBuffer(stream: Readable): Promise<Buffer> {
  const chunks: Buffer[] = [];
  for await (const chunk of stream) {
    chunks.push(Buffer.from(chunk));
  }
  return Buffer.concat(chunks);
}

In the modified code, I introduced a new streamToBuffer function that converts a ReadableStream to a Buffer. This function uses the for await...of loop to read the stream chunk by chunk and appends each chunk to an array of Buffer objects. Finally, it concatenates all the chunks into a single Buffer using Buffer.concat.

The streamToBuffer function is then used to convert the Response.body stream to a Buffer before passing it as the Body parameter of the PutObjectCommand.

Note that this approach reads the entire stream into memory before uploading it to S3. If you need to handle large files or streams efficiently, you might want to consider using the Upload utility provided by the @aws-sdk/lib-storage package which allows you to upload data in chunks without reading the entire stream into memory.

Hope it helps! Best, John

[simon-dk]

Only a bit related, but when passing an ArrayBuffer to the PutObjectCommand we also get a typeerror in 3.645.0.

It seems that the Smithy type StreamingBlobPayloadInputTypes not includes ArrayBuffer, even though the api accepts it.

Simple example below:

import type { File } from "node:buffer";

const upload = async (file: File) => {
  const buffer: ArrayBuffer = await file.arrayBuffer();

  const command = new PutObjectCommand({
    Bucket: "bucket",
    Key: `key`,
    Body: buffer, // Type 'ArrayBuffer' is not assignable to type 'StreamingBlobPayloadInputTypes | undefined
  });
};

Its easily fixable by casting to Buffer, but still something that might needs to be looked at.

const command = new PutObjectCommand({
    // ...
    Body: buffer as Buffer, // Works
  });
``

[Sophon96]

Hi @aBurmeseDev, thanks for the reply! Thanks for your solution -- I've also found that constructing a Uint8Array from response.arrayBuffer() also works (i.e. body: new Uint8Array(await response.arrayBuffer())). Thanks for the info about the lib-storage library; I wasn't aware of that.

However, are you aware of a method of narrowing the types so that type checking works? I'd rather not have to guess-and-check which types work. (I suppose this is also related to Simon's concern.)

Thanks again.

[aBurmeseDev]

Hey! Apologies for the long silence here. I'm glad you got it to work.

It seems that the Smithy type StreamingBlobPayloadInputTypes not includes ArrayBuffer, even though the api accepts it.

I believe this has been addressed in recent versions, if I'm not mistaken. Feel free to report back.