"ReferenceError: ReadableStream is not defined" from "@aws-sdk/lib-storage" on a stream pipe

[oprogramador]

Checkboxes for prior research

• [X] I've gone through Developer Guide and API reference
• [X] I've checked AWS Forums and StackOverflow.
• [X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

I have several issues related to uploading files into S3. Normally, it works even with large files but in some cases, it fails. This issue is easy to reproduce and looks to be definitely a bug inside the AWS SDK. Even if I do something wrong, I should get an appropriate error message.

SDK version number

"@aws-sdk/client-s3": "^3.350.0", "@aws-sdk/lib-storage": "^3.499.0",

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v16.19.0

Reproduction Steps

Run the following script with Node.js (node 1.js).

const Stream = require('stream');
const bluebird = require('bluebird');
const { Upload } = require('@aws-sdk/lib-storage');
const { S3 } = require('@aws-sdk/client-s3');

const run = async () => {
  const readable = new Stream.Readable({
    read() { },
  });
  const writable = Stream.Writable({
    write() {},
  });

  const params = {
    Bucket: 'my-bucket',
    Key: 'my-dir/pipe0.txt',
    Body: writable,
  };

  const uploadToS3 = new Upload({
    client: new S3(),
    queueSize: 4,
    partSize: 1024 * 1024 * 5,
    leavePartsOnError: true,
    params,
  });
  readable.pipe(writable);

  const result = readable.push('foo');
  console.log({ result });

  await bluebird.delay(5000);
  writable.end();
  await uploadToS3.done();
};

run()
  .then(() => console.log('done'))
  .catch((error) => console.error(error));

Observed Behavior

⟩ node 1.js 
{ result: true }
ReferenceError: ReadableStream is not defined
    at getChunk (/home/me/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:149:30)
    at _Upload.__doMultipartUpload (/home/me/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:361:24)
    at _Upload.done (/home/me/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:190:37)
    at run (/home/me/1.js:34:20)

Expected Behavior

A file should be uploaded into the S3 bucket (without any error).

Possible Solution

No response

Additional Information/Context

This script works (with just a file stream instead of anything from AWS SDK).

const fs = require('fs');
const bluebird = require('bluebird');

const run = async () => {
  const readable = new Stream.Readable({
    read() { },
  });
  const writable = fs.createWriteStream('./file-foo.txt');

  readable.pipe(writable);

  const result = readable.push('foo');
  console.log({ result });

  await bluebird.delay(5000);
  writable.end();
};

run()
  .then(() => console.log('done'))
  .catch((error) => console.error(error));

[oprogramador]

The following issues look to be related. However, I cannot find a pipe( example in any of them so IMO this issue should be kept open.

• https://github.com/aws/aws-sdk-js-v3/issues/3923
• https://github.com/aws/aws-sdk-js-v3/issues/2522
• https://github.com/aws/aws-sdk-js-v3/issues/2312
• https://github.com/aws/aws-sdk-js-v3/issues/3807
• https://github.com/aws/aws-sdk-js-v3/issues/3063

[major-winter]

Hi @oprogramador , are you sure that the Body is intented to refer to a function of type Writable? Since the Body accepts these types string | Uint8Array | Buffer | Readable

[oprogramador]

@major-winter

When passing the readable stream as a part of params instead of that writable stream, I get no error but no file appears on S3 either.

Anyway, if it doesn't accept a writable, a clear error message should be thrown.

[NorseGaud]

https://github.com/aws/aws-sdk-js-v3/issues/6013 related?

[aBurmeseDev]

Hi everyone on the thread - appreciate your patience.

You can resolve this by importing ReadableStream like so

import { ReadableStream } from 'web-streams-polyfill';

Let us know if issue persists and we'd be happy to investigate further.

[oprogramador]

@aBurmeseDev

After adding const { ReadableStream } = require('web-streams-polyfill');, this issue keeps happening.

[aBurmeseDev]

Can you share your updated code as well as the logs? Add this middleware stack to your client to retrieve request logs.

client.middlewareStack.add(
  (next, context) => async (args) => {
    console.log("AWS SDK context", context.clientName, context.commandName);
    console.log("AWS SDK request input", args.input);
    const result = await next(args);
    console.log("AWS SDK request output:", result.output);
    return result;
  },
  {
    name: "MyMiddleware",
    step: "build",
    override: true,
  }
);

[oprogramador]

You should be able to reproduce, using the details from this issue description.

This is resolved a long time ago on my side with a workaround.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.