Vendor our own fetch polyfill

[saghul]

The one we use now has some problems:

• Lack of streaming support (and the maintainer does not want to add it)
• Shenanigans as we polyfill it, since it detects support for things at import time

Generally speaking, it's small enough that it makes sense to incorporate it and make our own changes to it.

[Tobbe]

• Lack of streaming support (and the maintainer does not want to add it)

Am I right in assuming this is what you're referring to? https://github.com/JakeChampion/fetch/issues/198#issuecomment-494361060

Are there any other polyfills out there that could be a good fit for txiki.js? Or did you want to vendor JakeChampion/fetch and just add/modify what you need?

[saghul]

Am I right in assuming this is what you're referring to? JakeChampion/fetch#198 (comment)

Yes, pretty much.

Are there any other polyfills out there that could be a good fit for txiki.js? Or did you want to vendor JakeChampion/fetch and just add/modify what you need?

The latter, I think. I realized it's small enough that it should be easy to vendor and adapt, like remove all the support checks wich are done only once, since we know what the runtime supports.

[saghul]

Actually I've started to think that a better idea would be to drop XHR completely and implement fetch straight on top of libcurl.

If Node, Deno and Bun can get away with not having XHR we can too, I guess.

[lal12]

Actually I've started to think that a better idea would be to drop XHR completely and implement fetch straight on top of libcurl.

If Node, Deno and Bun can get away with not having XHR we can too, I guess.

Though there are some features not available in standard browser fetch that are possible with XHR. So some extensions are neeeded (nodejs e.g. has some too).

[saghul]

Do you have examples of those?

[lal12]

Two specifics I though of:

• Sending streams as body (available in chrome recently, not in FF or Safari)
  • also needed for progress of uploading (though some callback for this instead might also be useful)
• Controlling https policy to allow untrusted certificates e.g. (not available in browser at all)

I think being able to get the TCP connection from the fetch api (e.g. via Connection: Keep-alive or transport upgrade) to further use it would also be useful. Though this might raise other issues/questions when it is a SSL connection.

[saghul]

Good points!

I don't think we can extract the connection easily, since we don't use our own TCP socket, curl does...

[lal12]

Good points!

I don't think we can extract the connection easily, since we don't use our own TCP socket, curl does...

It's possible to either use CURLOPT_WRITEFUNCTION / CURLOPT_OPENSOCKETFUNCTION to supply your own socket. But that has issues for SSL. Though it also seems to be possible to get the socket from curl, and prevent it from closing (CURLOPT_CLOSESOCKETFUNCTION). Though I'm not sure how both would integrate with SSL.

[guest271314]

Make sure duplex: "half" and HTTP/2 are supported so we can full duplex stream.

[saghul]

Where can I find docs about the duplex option? I couldn't find anything on mdn.

[guest271314]

• WHATWG Fetch IDL Index at readonly attribute [RequestDuplex](https://fetch.spec.whatwg.org/#enumdef-requestduplex) [duplex](https://fetch.spec.whatwg.org/#dom-request-duplex);; and The new Request(input, init) constructor steps are:
• https://fetch.spec.whatwg.org/#dom-request-duplex
• https://fetch.spec.whatwg.org/#dom-requestinit-duplex
• duplex

"half" is the only valid value and it is for initiating a half-duplex fetch (i.e., the user agent sends the entire request before processing the response). "full" is reserved for future use, for initiating a full-duplex fetch (i.e., the user agent can process the response before sending the entire request). This member needs to be set when body is a ReadableStream object. See issue #1254 for defining "full".

• Streaming requests with the fetch API
• [Fetch body streams are not full duplex #1254] (https://github.com/whatwg/fetch/issues/1254) Referenced above
• Half duplex stream Full-duplex streaming using WHATWG fetch() in Chromium-based browsers between a ServiceWorker and WindowClient Node.js and Deno both support full-duplex streaming using fetch(). That is, we can upload the readable part of a TransformStream, write to the writable side of the same TransformStream, and read the readable side in the original single connection, ad infinitum. This is the only browser implementation of full-duplex streaming with WHATWG fetch() that I am aware of.
• Examples using Deno and Node.js Native Messaging hosts for their respective fetch() implementations to full-duplex stream from and to the browser with Web extensions native-messaging-deno fetch-duplex branch; native-messaging-nodejs full-duplex branch
• Related https://github.com/saghul/txiki.js/issues/450

Example client, in code, from https://github.com/oven-sh/bun/issues/7206

full_duplex_fetch_test.js full_duplex_fetch_test.js.zip

var wait = async (ms) => new Promise((r) => setTimeout(r, ms));
var encoder = new TextEncoder();
var decoder = new TextDecoder();
var { writable, readable } = new TransformStream();
var abortable = new AbortController();
var { signal } = abortable;
var writer = writable.getWriter();
var settings = { url: "https://comfortable-deer-52.deno.dev", method: "post" };
fetch(settings.url, {
  duplex: "half",
  method: settings.method,
  // Bun does not implement TextEncoderStream, TextDecoderStream
  body: readable.pipeThrough(
    new TransformStream({
      transform(value, c) {
        c.enqueue(encoder.encode(value));
      },
    }),
  ),
  signal,
})
  // .then((r) => r.body.pipeThrough(new TextDecoderStream()))
  .then((r) =>
    r.body.pipeTo(
      new WritableStream({
        async start() {
          this.now = performance.now();
          console.log(this.now);
          return;
        },
        async write(value) {
          console.log(decoder.decode(value));
        },
        async close() {
          console.log("Stream closed");
        },
        async abort(reason) {
          const now = ((performance.now() - this.now) / 1000) / 60;
          console.log({ reason });
        },
      }),
    )
  ).catch(async (e) => {
    console.log(e);
  });
await wait(1000);
await writer.write("test");
await wait(1500);
await writer.write("test, again");
await writer.close();

Deno server code on Deno Deploy that echoes to the Native Messaging extension, and public requests, i.e., from /to the client above

Deno Deploy full-duplex server

const responseInit = {
  headers: {
    'Cache-Control': 'no-cache',
    'Content-Type': 'text/plain; charset=UTF-8',
    'Cross-Origin-Opener-Policy': 'unsafe-none',
    'Cross-Origin-Embedder-Policy': 'unsafe-none',
    'Access-Control-Allow-Origin': '*',
    'Access-Control-Allow-Private-Network': 'true',
    'Access-Control-Allow-Headers': 'Access-Control-Request-Private-Network',
    'Access-Control-Allow-Methods': 'OPTIONS,POST,GET,HEAD,QUERY',
  },
};

for await (
  const conn of Deno.listen({
    alpnProtocols: ["h2", "http/1.1"],
  })
) {
  for await (const {
      request,
      respondWith
    }
    of Deno.serveHttp(conn)) {
    if (request.method === 'OPTIONS' || request.method === 'HEAD') {
      respondWith(new Response(null, responseInit));
    }

    if (request.method === 'GET') {
      respondWith(new Response(null, responseInit));
    }
    try {
      const stream = request.body
        .pipeThrough(new TextDecoderStream())
        .pipeThrough(
          new TransformStream({
            transform(value, c) {
              c.enqueue(value.toUpperCase());
            },
            async flush() {

            },
          })
        ).pipeThrough(new TextEncoderStream());
      respondWith(new Response(
        stream, responseInit));
    } catch (e) {

    }
  }
}

The above is a single HTTP/2 connection. The closest we get to that in HTTP 1.1 is WebSocketStream. The excpetion, as noted above, is between a ServiceWorker and a WindowClient on Chromium-based browsers. I've asked. Not sure how Chromium authors implement that particular connection.

Expected result

$ deno run -A full_duplex_fetch_test.js
495.081126
TEST
TEST, AGAIN
Stream closed

$ node --experimental-default-type=module full_duplex_fetch_test.js
1286.0589990019798
TEST
TEST, AGAIN
Stream closed

What happens in Bun, that does not, yet, implement HTTP/2

$ bun run full_duplex_fetch_test.js
24112.282304
Stream closed

[saghul]

Thanks!

[saghul]

The fetch polyfill is vendored as of https://github.com/saghul/txiki.js/pull/647

I laid out the plan for the future in that PR.

I'm closing this one since the initial goal (simple vendoring of the polyfill) is done, but will be coming back for the useful comments.

[guest271314]

tjs run full-duplex-fetch-test.js
Error: Unsupported payload type
    at send (polyfills.js:2:37741)
    at <anonymous> (polyfills.js:2:3115)
    at Promise (native)
    at k (polyfills.js:2:3115)
    at <anonymous> (full-duplex-fetch-test.js:21:1)
    at evalFile (native)
    at <anonymous> (run-main.js:27:1435)

Maybe try to use Workerd, Deno, or Node.js implementation of WHATWG Fetch and WHATWG Streams?

[saghul]

The change needs to happen in the native implementation now, and I cannot borrow that.

[guest271314]

Doesn't txiki.js depend on cURL? cURL supports HTTP/2.

[saghul]

Yes, but the code needs to be written to send data to the native side in chunks in case it's a ReadableStream.