node icon indicating copy to clipboard operation
node copied to clipboard

lib: add support for readable byte streams to .toWeb()

Open seriousme opened this issue 5 months ago β€’ 2 comments

Add support for the creation of ReadableByteStream to Readable.toWeb() and Duplex.toWeb() This enables the use of .getReader({ mode: "byob" }) on e.g. socket().toWeb()

e.g.:

 const { writable, readable } = Duplex.toWeb(duplex, { type: 'bytes' });
 const data = new Uint8Array(dataToRead.length);
 readable.getReader({ mode: 'byob' }).read(data).then((result) => {
    // do something with result
 });

Kind regards, Hans

Refs: https://github.com/nodejs/node/issues/56004#issuecomment-2908265316 Refs: https://developer.mozilla.org/en-US/docs/Web/API/Streams_API/Using_readable_byte_streams

seriousme avatar Jun 10 '25 18:06 seriousme

Review requested:

  • [ ] @nodejs/streams

nodejs-github-bot avatar Jun 10 '25 18:06 nodejs-github-bot

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 89.95%. Comparing base (09b4c57) to head (55af65e). :warning: Report is 916 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #58664      +/-   ##
==========================================
- Coverage   90.07%   89.95%   -0.13%     
==========================================
  Files         641      667      +26     
  Lines      188998   196801    +7803     
  Branches    37069    38420    +1351     
==========================================
+ Hits       170246   177023    +6777     
- Misses      11462    12205     +743     
- Partials     7290     7573     +283     
Files with missing lines Coverage Ξ”
lib/internal/streams/duplex.js 100.00% <100.00%> (ΓΈ)
lib/internal/webstreams/adapters.js 85.59% <100.00%> (+0.36%) :arrow_up:

... and 224 files with indirect coverage changes

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codecov[bot] avatar Jun 14 '25 09:06 codecov[bot]

this lgtm - only thing missing would be a test when the user provides an invalid type argument value.

Thx, just added the test on the validity of the type option.

seriousme avatar Jun 16 '25 14:06 seriousme

CI: https://ci.nodejs.org/job/node-test-pull-request/67499/

nodejs-github-bot avatar Jun 17 '25 13:06 nodejs-github-bot

CI: https://ci.nodejs.org/job/node-test-pull-request/67524/

nodejs-github-bot avatar Jun 18 '25 18:06 nodejs-github-bot

CI failures seem unrelated to me

Ethan-Arrowood avatar Jun 18 '25 19:06 Ethan-Arrowood

CI: https://ci.nodejs.org/job/node-test-pull-request/67576/

nodejs-github-bot avatar Jun 20 '25 17:06 nodejs-github-bot

CI: https://ci.nodejs.org/job/node-test-pull-request/67591/

nodejs-github-bot avatar Jun 21 '25 15:06 nodejs-github-bot

I applied the requested changes.

seriousme avatar Jun 27 '25 17:06 seriousme

Sorry for the noob question, but do I resolve conflicts in doc/api/stream.md or will the process handle this by itself ? If I need to do this, what is the best way to do this without messing up the git branch ?

seriousme avatar Jun 27 '25 17:06 seriousme

@seriousme your best option is to mergen main into this branch. Alternatively, you can do a git rebase. Before doing anything anyway, backup the current branch in another.

mcollina avatar Jun 28 '25 20:06 mcollina

Ok, I merged upstream/main.

seriousme avatar Jun 29 '25 07:06 seriousme

@jasnell @mcollina are you both ok with the changes applied to this PR or is there anything more left to do?

seriousme avatar Jul 06 '25 11:07 seriousme

Could someone please start CI again? I am assuming no other changes are required for now.

seriousme avatar Jul 09 '25 15:07 seriousme

Failed to start CI
   ⚠  Commits were pushed since the last approving review:
   ⚠  - lib: add support for readable byte streams to .toWeb()
   ⚠  - fix: correct link from webstream to streamduplextowebstreamduplex
   ⚠  - rewrite of test-stream-readable-to-web-byob.js
   ⚠  - add validation of 'type' option
   ⚠  - use internal validators to validate options
   ⚠  - Update lib/internal/webstreams/adapters.js
   ⚠  - add item to doc changes
   ⚠  - applied various changes
   ⚠  - fixed test-stream-readable-to-web-byob.js
   ⚠  - applied review changes to adapters.js
   ⚠  - Merge remote-tracking branch 'upstream/main' into lib-add-toweb-reada…
   ⚠  - remove colon from description as it breaks the YAML
   ⚠  - add  single quotes to 'type'
   ✘  Refusing to run CI on potentially unsafe PR
https://github.com/nodejs/node/actions/runs/16175195572

github-actions[bot] avatar Jul 09 '25 16:07 github-actions[bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/67913/

nodejs-github-bot avatar Jul 09 '25 16:07 nodejs-github-bot

I will try to do the rebase tomorrow.

seriousme avatar Jul 09 '25 19:07 seriousme

Ok, rebased. Can someone start CI again?

seriousme avatar Jul 10 '25 10:07 seriousme

CI: https://ci.nodejs.org/job/node-test-pull-request/67920/

nodejs-github-bot avatar Jul 10 '25 11:07 nodejs-github-bot

CI seems to have passed al checks, can someone please decide to merge or let me know what needs to be changed. I'd rather not have to rebase again just because this PR is sitting idle.

seriousme avatar Jul 10 '25 15:07 seriousme

CI: https://ci.nodejs.org/job/node-test-pull-request/67926/

nodejs-github-bot avatar Jul 10 '25 23:07 nodejs-github-bot

I checked and last nights build failures do not seem related to this PR.

  • RHEL 8 on s390 (where rhel 9 on s390 works)
  • win11-arm64-COMPILED_BY-vs2022_clang-arm64 (where all other windows builds incl other arm64 works)

Btw: I read in the contribution guide:

If you see that requested changes have been made, you can clear another collaborator's Changes requested review.

Maybe someone can verify that the requested changes were applied? (as far as I can see I followed up on all of them)

seriousme avatar Jul 11 '25 05:07 seriousme

@mcollina could you please check if everything is ok?

seriousme avatar Jul 16 '25 17:07 seriousme

@Ethan-Arrowood @jasnell @MattiasBuelens @mcollina You all spent time providing valueable feedback, thanks for that! I think it would be sad if you spent that precious time without seeing a NodeJS improvement as result. As far as I can tell all your wishes have been implemented, so we just need someone to approve and for obvious reasons that can't be me!

Kind regards, Hans ps. if anything still needs to be done I'm more than happy to help out!

seriousme avatar Jul 18 '25 15:07 seriousme

CI: https://ci.nodejs.org/job/node-test-pull-request/68019/

nodejs-github-bot avatar Jul 18 '25 15:07 nodejs-github-bot

CI: https://ci.nodejs.org/job/node-test-pull-request/68021/

nodejs-github-bot avatar Jul 18 '25 15:07 nodejs-github-bot

Given all the work that you already have done, is there anything we can do to land this PR?

seriousme avatar Jul 22 '25 18:07 seriousme

@Ethan-Arrowood @jasnell @MattiasBuelens @mcollina

It feels to me like we are almost there and you all put so much time in it that I think it would be sad to let this PR just slip into oblivion. @mcollina can you please review? If not can you please let us know? If you don't have the time to review, would it be ok if someone else signs off that I implemented your requested changes?

Thanks again for all the effort!

Kind regards, Hans

seriousme avatar Jul 30 '25 17:07 seriousme

@mcollina @jasnell @Ethan-Arrowood @MattiasBuelens

Fellow NodeJS enthousiasts, I appreciate that you are all very busy improving NodeJS! What can we do to land this PR and thereby land another improvement? (which someone refered to as "Solid work" πŸ˜‰ )

Kind regards, Hans ps. I don't need credits or my name on any list or whatever, I just would like to fix this use case so that I can cleanup my own code and help others in the process, in the true spirit of Open Source.

seriousme avatar Sep 01 '25 14:09 seriousme

@seriousme please address @jasnell comments

mcollina avatar Sep 01 '25 17:09 mcollina