ReferenceError: ReadableStream is not defined (lib-storage chunker error on Node.js)

[garrett-hopper]

Sorry for the noise, but can #2312 be reopened? It was closed and later locked with a mentioned of no "recent activity" despite there being a couple comments about it not being resolved. Was this a bug in the GitHub action not seeing the following comments?

[ajredniwja]

Hi @garrett-hopper thanks for brining this is up, It was not a bug with GH actions, the issue was closed intially because of lack of response, which is set to 10 days, then the another GH action locks the thread with no activity for 14 days which was after your comment, we tend to lock the threads since they are not so visible after being locked and we encourage to open new issues if thats the case.

With that being said, I have re-opened the issue but it seems like Alex was not able to reproduce the issue, can you try running his code?

I'll take more questions on other issue, feel free to tag me or anything for any questions.

[garrett-hopper]

Ah, that makes sense. Thanks!

[garrett-hopper]

It looks like #2312 is still locked, so I can't comment there. I'm not sure about @alexforsyth not being able to reproduce the original issue, as the chunker at lib-storage/src/chunker.ts#L22 definitely doesn't handle ReadableStream not being defined on Node. Nor does it handle objects which match the required spec but don't actually contain any of the checked types in its prototype chain for the instanceof checked.

npm i @aws-sdk/lib-storage tar
node -e "require('@aws-sdk/lib-storage/dist/cjs/chunker').getChunk(require('tar').c({gzip: true}, ['test']))"

/home/garrett/tmp/node_modules/@aws-sdk/lib-storage/dist/cjs/chunker.js:25
    else if (data instanceof ReadableStream) {
                             ^

ReferenceError: ReadableStream is not defined
    at Object.getChunk (/home/garrett/tmp/node_modules/@aws-sdk/lib-storage/dist/cjs/chunker.js:25:30)
    at [eval]:1:50
    at Script.runInThisContext (node:vm:129:12)
    at Object.runInThisContext (node:vm:305:38)
    at node:internal/process/execution:81:19
    at [eval]-wrapper:6:22
    at evalScript (node:internal/process/execution:80:60)
    at node:internal/main/eval_string:27:3

[garrett-hopper]

I ended up switching from jszip to archiver which provided a proper Node Readable instance. There's likely others out there which will attempt to use similar libraries which don't provide Node Readable instances, and the ReadableStream browser API not being defined in Node is an annoying red herring to track down. Ideally it would support some of these less conventional formats, but at the very least, it should fall down to lib-storage/src/chunker.ts#L26 and provide a proper error message.

[ajredniwja]

I'll close the other one and keep this one opened.

[mifi]

I'm also seeing this error with streams created by into-stream. Maybe we can take inspiration from this module? https://github.com/sindresorhus/is-stream/blob/main/index.js

[mifi]

A workaround is to pipe the stream through a PassThrough stream:

const { PassThrough, pipeline } = require('stream');

const passThrough = new PassThrough();
pipeline(stream, passThrough, () => {});

const upload = new Upload({
  client: new S3({}),
  params: {
    Bucket: '...',
    Key: '...',
    Body: passThrough,
  },
  leavePartsOnError: true,
});

[ByScripts]

I have the exact same problem.

I'm trying to upgrade SDK from v2 to v3.

v2:

await this._s3.upload(
  {
    Bucket: <a>,
    Key: <b>,
    Body: <c>,
  },
  {
    partSize: <d>,
    queueSize: <e>
  }
)

v3:

const upload = new Upload({
  client: this._s3,
  params: {
    Bucket: <a>,
    Key: <b>,
    Body: <c>,
  },
  partSize: <d>,
  queueSize: <e>,
});

await upload.done();

But I get the error:

ReferenceError: ReadableStream is not defined
    at Object.getChunk (node_modules/@aws-sdk/lib-storage/dist/cjs/chunker.js:25:30)`

Thanks @mifi for the workaround. It seems it works.

[ajredniwja]

Related: https://github.com/aws/aws-sdk-js-v3/issues/1877#issuecomment-755387549

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.