[v2.0.0][Node.js][stream] `uploadFile` stalls with 8MB file, uploading enclosing directory with `uploadFilesFromDirectory` succeeds

[Cafe137]

I have not debugged it much yet, but I encountered this issue in swarm-cli.

I have an 8MB binary file for testing purposes. When I try to upload it, the bee.uploadFile never settles.

When I upload the directory containing the 8MB binary file as well as some other smaller files, the operation succeeds (bee.uploadFilesFromDirectory).

curl -X POST http://localhost:1633/bzz?name=8mb.bin -H "accept: application/json, text/plain, */*" -H "content-length: 8000000" -H "content-type: application/octet-stream" -H "swarm-postage-batch-id: 6f4622d44519709f66ae1e6c52f2000c232e30cb867713a983660e49c85a5552" -H "user-agent: bee-js/2.0.0"

[Cafe137]

I am uploading to an Ubuntu 20.04 VPS with Bee-Factory running 1.1 with one queen and one worker.

[Cafe137]

Also there are no logs in Bee indicating that anything is happening at all

[AuHau]

Thanks for the report. Will investigate!

[Cafe137]

First I tried uploading RANDOM uncompressable 8MB -> I cancelled because it gets nowhere

Then put it in a dir, uploaded dir -> succeeds very shortly

Uploaded single file README.md which is small -> succeeds immediately

If I upload it via curl, it succeeds very shortly

curl -X POST "http://localhost:1633/bzz?name=8mb.bin" -H "accept: application/json, text/plain, */*" -H "content-type: application/octet-stream" -H "swarm-postage-batch-id: d535b3315354becdc7d843ad7b9f6ff1dd31ae23a99906cbff491da2b0cfd13a" -H "user-agent: bee-js/2.0.0" --data @rand/8mb.bin

[Cafe137]

I tried to get a simple ky POST working with ReadStream, but ky expects ReadableStream - the two are not compatible.

import { createReadStream } from 'fs'
import ky from 'ky-universal'

const stream = createReadStream('chunk.bin')

const instance = ky.create({
  prefixUrl: 'http://localhost:1633',
  timeout: false,
})

instance.post('bzz?name=chunk.bin', {
  headers: {
    'swarm-postage-batch-id': 'd535b3315354becdc7d843ad7b9f6ff1dd31ae23a99906cbff491da2b0cfd13a',
    'content-length': '8000000',
    'content-type': 'application/octet-stream',
  },
  body: stream,
})

[AuHau]

Well, did it really errored out? Or what is exactly the problem? Types?

ky is typed for the WHATWG ReadableStream but on NodeJS it uses node-fetch which supports Readable (and hence ReadStream which is extended from Readable) and hence the types are not really correct.

[Cafe137]

Yeah so TypeScript kept yelling at me that the types are not compatible, but when I ran it anyways (maybe a slightly modified version of the code) I ran into a runtime error too. Bee-js 2.0 works well with Buffer instead ReadStream, so I am focusing on getting my PR ready that way, then I can continue investigating this issue.

[AuHau]

Hmmm, honestly I think there some other problem as it should work and it works currently in the Bee-js. If ky would not be able to read the stream, not even the first chunk would leave (for me it was actually multiple chunks)...

But yeah, let's do the workaround as discussed, only please don't use Buffer as you would to polyfill it for browsers. Or you can use it for NodeJS part, but not in browsers if you think, but generally I guess better would be to use Uint8Array (I guess you will have to employ Blob for that as well).