undici
undici copied to clipboard
nodejs
Fetch highWaterMark
What should we use as highWaterMark for the fetch api? I think browsers usually have 1m+ while Node has ~16k. Should we try matching the browsers? I think this has some implications on functionality when using Response.clone.
Can it be updated dynamically? E.g when the response is not cloned, then it can be 64kb. If someone reads from the clone then increase it again by 64kb if it's been read.
But if it's only 64k I think we could just have it as a default. It's more relevant if we have something like > 128k.
https://github.com/node-fetch/node-fetch/issues/151 https://github.com/node-fetch/node-fetch/issues/248
But if it's only 64k I think we could just have it as a default. It's more relevant if we have something like > 128k.
Performance-wise I don't think there's a difference.
because we are not reading from cloned stream at the time, its internal buffer will fill up, eventually, causing both stream to pause and hence what we called backpressure.
I mean if the response is cloned, then every time it's consumed bit by bit increase the highWaterMark so it can flow.
Example:
- A new response is created.
- It's being cloned.
- 64k is read from the clone. In order not to pause the stream, increase the
highWaterMarkby 64k. - The
highWaterMarkis now 128k. - Another 64k is read from the clone. In order not to pause the stream, increase the
highWaterMarkby 64kb. - The
highWaterMarkis now 192k. - 192k is read from the original response. In order not to pause the stream, increase
highWaterMarkagain by 64k. - The
highWaterMarkis now 256k.
Take a look at https://www.npmjs.com/package/cloneable-readable on how I clone Node.js streams. In our world I did that by setting the highwaterMark of the clone, and then the pipe mechanism did the rest.
