Hosting rewrite post call to function times out

[jkibble]

[REQUIRED] Environment info

firebase-tools: 12.3.1

Platform: macOS

[REQUIRED] Test case

https://github.com/jkibble/firebase-sveltekit-broken-rewrite

[REQUIRED] Steps to reproduce

All default steps with added a rewrite entry in firebase.json, however I created a repo which demonstrates the problem https://github.com/jkibble/firebase-sveltekit-broken-rewrite

npx firebase emulators:start navigate to http://localhost:5000 click the Get call click the Post call

[REQUIRED] Expected behavior

Post calls succeeds

[REQUIRED] Actual behavior

Get call succeeds Post call times out with 504 gateway error without running any server side code

[google-oss-bot]

This issue does not have all the information required by the template. Looks like you forgot to fill out some sections. Please update the issue with more information.

[jkibble]

After more investigation the rewrite was completely ignored in 12.2.1 so I suspect #5923 had something to do with the making work. It also hits the svelte ssr function first which returns a 404 then calls the api function according to the logs. Not sure if that's expected behaviour or not.

Running firebase emulators:exec "vite dev" does actually work as expected but I suspect it's running in static mode at this point so there is no ssr.

[jkibble]

even more digging and it appears node-fetch package is the end of the line. The timeout is because, I believe, the content-length longer than expected and it times out waiting for the body to match the content-length.

Hard coding the content-length in node_modules/firebase-tools/lib/hosting/proxy.js:73 to 0 causes the fetch to complete, anything higher caused it to timeout. Not sure where the content-length header is set but it looks correct to me, the issue is probably the POST body is empty by this point.

[jkibble]

If I hard code the hostname to 127.0.0.1, port to 5001 and path to whatever the redirect is after the 404 it all works as expected. https://github.com/firebase/firebase-tools/blob/3f3c1868ef5adce4cc9217fbe54c54ef31b01433/src/frameworks/utils.ts#L105

Somewhere along the way the process of sending the request first to the sveltekit ssr function and running next() the request body doesn't appear to be present anymore. It's, I believe, a readable stream at this point so I'd assume reading it would advance the pointer and consequently it's empty by the time the next() is called on line 118

[jpenna]

This is happening for me as well. I'm using [email protected]. I've tried with both Node 16 and 18.

I'm using next.js and Firebase's webframeworks is activated. I separated my functions from Next.js because they were not deployed correctly, even though they worked with the emulator. Now I can't call the functions with the emulator because they timeout (in production I didn't test).

When I call my API with the resolved URL (http://127.0.0.1:5001/project-e9500/southamerica-east1/createProfile/api/profile) it works as expected, but when I try to use the hosting redirect (http://localhost:3000/api/profile), it never calls the function and times out after 1 minute.

After looking into it, I found that removing fetchOptions.body from the request in apiv2.js (code makes it go through. fetchOptions.body is a pipped stream containing the complete request body (proxy.js code).

---

A partial section of the firebase.json:

  "hosting": {
    "source": "web",
    "ignore": ["firebase.json", "**/.*", "**/node_modules/**", "**/coverage/**"],
    "frameworksBackend": {
      "region": "us-east1",
      "maxInstances": 10
    },
    "rewrites": [
      {
        "source": "/api/profile",
        "function": {
          "functionId": "createProfile",
          "region": "southamerica-east1",
          "pinTag": true
        }
      },
....

[email protected] skips redirect and forwards directly to Next.js framework.

[email protected] worked when calling the function's path directly, but printed this error after calling the "hosting redirect url":

Error [ERR_STREAM_CANNOT_PIPE]: Cannot pipe, not readable
    at new NodeError (node:internal/errors:387:5)
    at ServerResponse.pipe (node:_http_outgoing:1115:22)
    at /home/juliano/.nvm/versions/node/v16.20.2/lib/node_modules/firebase-tools/lib/frameworks/utils.js:47:18
    at /home/juliano/.nvm/versions/node/v16.20.2/lib/node_modules/firebase-tools/lib/frameworks/utils.js:96:45
i  hosting: 127.0.0.1 - - [04/Sep/2023:21:30:39 +0000] "GET /api/profile HTTP/1.1" 500 - "http://localhost:3000/profile" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36" {"metadata":{"emulator":{"name":"hosting"},"message":"127.0.0.1 - - [04/Sep/2023:21:30:39 +0000] \"GET /api/profile HTTP/1.1\" 500 - \"http://localhost:3000/profile\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36\""}}
Error: socket hang up
    at connResetException (node:internal/errors:705:14)
    at Socket.socketCloseListener (node:_http_client:467:25)
    at Socket.emit (node:events:525:35)
    at Socket.emit (node:domain:489:12)
    at TCP.<anonymous> (node:net:301:12) {
  code: 'ECONNRESET'
}

[email protected]+ times out.

[jkibble]

Excellent find. I spent a few hours trying to track it down but digging around the code in node_modules was very challenging.

For what it's worth it does work in production, at least for me, but I was unwilling to trust it since I couldn't test the code locally

[jpenna]

I've spent the last 2 days trying to fix this, but failed. I don't want to spend more time on this, so I did a quickfix for my use case: https://github.com/firebase/firebase-tools/compare/master...jpenna:firebase-tools:api-jp

This seems like a very important fix, because it blocks the usage of NextJS with standalone Functions.

---

Findings

Like I said before, removing the body from the request in apiv2.ts (commenting out this line) sends back a response (didn't timeout), but then we ignore the body, which is unnacceptable.

Calling next() after this line (similar to what I'm doing in my adhoc solution) works for the API, but fails if it is meant for NextJS (like navigation). That's why I added the if (req.url?.startsWith("/api")), so only the calls to /api will call next() and, since I know I don't care about the framework at this path, I skipped the resolution of NextJS.

It seems like Next's request handler is being called for all requests as a middleware set in the server. There is a logic that calls the next middleware when the framework's resolver returns 404 (code). It seems like the timeout isn't related to the following, but if you try to change the headers before calling next() (I tried to reset the headers in proxiedRes), then it fails with "cannot change headers already sent" because of a call to proxiedRes.writeHead (where is it coming from?).

Hopefully someone with more knowledge of Superstatic and NextJS will fix it easily.

@jamesdaniels since you were one of the last people touching this area, would you know how to fix it?

[triplef]

We’re seeing the same since integrating Next.js. Also happens only when using the emulator, calls to production work fine.

However we found that using the direct function URL without the hosting redirect works correctly:

• http://localhost:5000/api/endpoint timeout
• http://localhost:5001/<project>/us-central1/api/endpoint works

[ruslan-simonenko]

I am still experiencing this issue with React + Web-Frameworks & Firebase Tools: 13.3.1. Can't manage to make hosting rewrites work with emulators.

I suspect https://github.com/firebase/firebase-tools/pull/5923 had something to do with the making work. It also hits the svelte ssr function first which returns a 404 then calls the api function according to the logs. Not sure if that's expected behaviour or not.

I believe rewrite should kick in before the web app, I assume this is a bug.

I resorted to vite proxy for rewrites in the meantime:

server: {
      proxy: {
        '/handle-card': {
          target:
            `http://127.0.0.1:5001/${projectID}/${projectRegion}/card-redirect`,
          changeOrigin: true,
          rewrite: (path) => path.replace(/^\/handle-card/, ''),
        },

Unfortunately this means that I can't verify firebase hosting rewrites without deploying and manually testing.