Does fetch support piping body into request?

[rysi3k]

Hello, I'm trying to create proxy which pass all incoming data into fetch request to an another server, but every try causes error/no success. Could someone help me? I'm trying do something like that:

router.post('/test', async (req, res) => {
        var r = new stream.PassThrough();
        req.pipe(r);
        await fetch('http://some-url', {
            body: stream.Readable.from(r),
            method: 'POST',
            headers: req.headers,
        });
        res.sendStatus(202);
    });

or

router.post('/test', async (req, res) => {
        await fetch('http://some-url', {
            body: req,
            method: 'POST',
            headers: req.headers,
        });
        res.sendStatus(202);
    });

I need to achieve the same result, as old lib does: https://www.npmjs.com/package/request#streaming

I'm using node 18 and embedded fetch method.

Thanks in advance. Regards

[jimmywarting]

this is the kind of resons i would like to see IncomingMessage.request and respondWith being supported in nodes built in http library...

anyhow try something like this:

router.post('/test', async (r, res) => {
  const ctrl = new AbortController()
  const headers = new Headers(r.headers)
  const url = `...`

  r.once('aborted', () => ctrl.abort())
 
  const req = new Request(url, {
    headers,
    method: r.method,
    body: r.method === 'GET' || r.method === 'HEAD' ? null : r,
    signal: ctrl.signal,
    referrer: headers.get(referrer)
  })
  fetch(request).then(response => {
    stream.Readable.fromWeb(response.body).pipe(res)
  })
})

[rysi3k]

Nah, when using:

body: r.method === 'GET' || r.method === 'HEAD' ? null : r,

I've got error: TypeError: Response body object should not be disturbed or locked at extractBody (node:internal/deps/undici/undici:2140:17) at new Request (node:internal/deps/undici/undici:5626:48)

After change to:

body: r.method === 'GET' || r.method === 'HEAD' ? null : stream.Readable.from(r),

got: TypeError: fetch failed RequestContentLengthMismatchError: Request body length does not match content-length header

[jimmywarting]

😅 just coded blindly without actually testing

if the response is content-encoded then you would need to remove that headers as well as content-length headers as fetch would decode the response (hence why i also made this request)

[rysi3k]

Hmm I'm testing using only curl, without compressing, so it is plain text imo.

curl -v -i -X POST \
   -H "Content-Type:application/json" \
   -d \
'{"foo":"3"}' \           
 'http://127.0.0.1:3000/test'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.1:3000...
* Connected to 127.0.0.1 (127.0.0.1) port 3000 (#0)
> POST /test HTTP/1.1
> Host: 127.0.0.1:3000
> User-Agent: curl/7.79.1
> Accept: */*
> Content-Type:application/json
> Content-Length: 11
> 
^C

Content-length=11 equals payload size.

My endpoint:

    router.post('/test', async (r, res) => {
        const ctrl = new AbortController();
        delete r.headers.connection;
        const headers = new Headers(r.headers);
        const url = 'https://.....';

        r.once('aborted', () => ctrl.abort());

        const req = new Request(url, {
            headers,
            method: r.method,
            body: r.method === 'GET' || r.method === 'HEAD' ? null : stream.Readable.from(r),
            signal: ctrl.signal,
        })
        fetch(req).then(response => {
            console.log(response)
          // stream.Readable.fromWeb(response.body).pipe(res)
        });
    });

What I'm doing wrong?

Regards

[AlexRMU]

Example of a simple proxy for node 18+ https://proxy.site:8080/?url=https://target.site/

import http from "http";
import stream from "stream";

const forbidden_req_headers = ["host"];
const forbidden_res_headers = [
    "set-cookie",
    "content-length",
    "content-encoding",
    "transfer-encoding",
    "content-security-policy-report-only",
    "content-security-policy",
];

http.createServer(async (req, res) => {
    const url = new URL(req.url, "https://" + req.headers.host).searchParams.get("url") || "";
    console.log("url", url);

    const req_headers = Object.fromEntries(
        Object.entries(req.headers).filter(
            ([k, v]) => !forbidden_req_headers.includes(k.toLowerCase())
        )
    );
    // console.log("req_headers", req_headers);
    req.headers = req_headers;

    const controller = new AbortController();
    req.once("aborted", () => controller.abort());
    const request = new Request(url, {
        headers: new Headers(req.headers),
        method: req.method,
        body: req.method == "GET" || req.method == "HEAD" ? undefined : req,
        signal: controller.signal,
        referrer: req.headers.referrer,
        duplex: "half",
    });
    await fetch(request)
        .then(async (response) => {
            const headers = Object.fromEntries(response.headers);
            const res_headers = Object.fromEntries(
                Object.entries({
                    ...headers,
                    "Access-Control-Allow-Origin": "*",
                    "Access-Control-Allow-Headers": "*",
                    "Access-Control-Expose-Headers": "*",
                    "Access-Control-Allow-Methods": "*",
                    "Access-Control-Allow-Credentials": "true",
                }).filter(([k, v]) => !forbidden_res_headers.includes(k.toLowerCase()))
            );
            // console.log("res_headers", res_headers);
            for (const key in res_headers) {
                res.setHeader(key, res_headers[key]);
            }

            stream.Readable.fromWeb(response.body).pipe(res);
        })
        .catch((error) => {
            console.log({ error });
            res.writeHead(500);
            res.end(JSON.stringify({ res: error + "" }));
        });
}).listen(8080);

[jimmywarting]

@AlexRMU just beware for someone doing things like https://proxy.site:8080/?url=http://localhost or if undici is ever going to support file:// protocol which they might do in the feature - at least there have been talks about for and against it in the past. then u also need to be aware of things like https://proxy.site:8080/?url=file:///etc/passwd

Both Deno & Bun do support fetch('file:///...') so i would guess that there isn't so unlikely that undici will also.

i have personally built a cors proxy myself in the good old days using node-fetch.

[RaeesBhatti]

Example of a simple proxy for node 18+ https://proxy.site:8080/?url=https://target.site/

import http from "http";
import stream from "stream";
...
    const request = new Request(url, {
        headers: new Headers(req.headers),
        method: req.method,
        body: req.method == "GET" || req.method == "HEAD" ? undefined : req,
        signal: controller.signal,
        referrer: req.headers.referrer,
        duplex: "half",
    });
...

This works fine for requests without a body but requests with a body throw the following error:

node:internal/deps/undici/undici:12618
    Error.captureStackTrace(err, this);
          ^
DOMException [AbortError]: This operation was aborted
    at node:internal/deps/undici/undici:12618:11
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

You can do something like:

const request = new Request(url, {
    headers: new Headers(req.headers),
    method: req.method,
    body: req.method == "GET" || req.method == "HEAD" ? undefined : req.socket,
    signal: controller.signal,
    referrer: req.headers.referrer,
    duplex: "half",
});

This doesn't result in an error but any request with a body will just hang and timeout.

[RaeesBhatti]

The reason the requests hang is because fetch waits for the request body stream to end at some point. Fetch keeps waiting for the stream to end but Node.js keeps the req.socket open, which results in a timeout.

I came up with the following solution. You might need to modify it to fit your needs.

let clientRequestBody: ReadableStream | undefined = undefined;

if (this.httpMethodCanHaveBody(clientRequestMethod)) {
  const contentLength = clientRequestHeaders.get('content-length');
  if (contentLength === null) {
    clientResponse.writeHead(411, 'Length Required');
    clientResponse.end();
    return;
  }

  const contentLengthValue = Number.parseInt(contentLength, 10);
  if (Number.isNaN(contentLengthValue)) {
    clientResponse.writeHead(400, 'Bad Request');
    clientResponse.end();
    return;
  }

  const clientRequestBodyIterator = this.getStreamIterator(clientRequest);

  // create a ReadableStream from the clientRequestBodyReader that reads until the contentLengthValue
  const bodyReader = new ReadableStream({
    async start(controller) {
      let totalBytesRead = 0;
      do {
        const { done, value } = await clientRequestBodyIterator.next();
        if (value) {
          totalBytesRead += value.length;
          if (totalBytesRead > contentLengthValue) {
            clientResponse.writeHead(400, 'Bad Request');
            clientResponse.end();
            return;
          }

          controller.enqueue(value);

          if (totalBytesRead === contentLengthValue) {
            controller.close();
            break;
          }
        }

        if (done) {
          break;
        }
      } while (true);
    },
  });

  // @ts-ignore
  clientRequestBody = ReadableStream.from(contentLengthValue === 0 ? [] : bodyReader);
}

async* getStreamIterator(stream: Readable) {
  for await (const chunk of stream) {
    yield chunk;
  }

  return undefined;
}