initializeTestEnvironment sometimes failed due to Node.js fetch failed by SocketError: other side closed

[jin-qin]

Operating System

Ubuntu 22.04

Environment (if applicable)

GitHub Actions

Firebase SDK Version

11.1.0

Firebase SDK Product(s)

Firestore

Project Tooling

React Native (Expo), Jest, TypeScript.

Detailed Problem Description

Hi Firebase team,

We recently occasionally have this error in our GitHub workflow, we've never had this issue in local device:

    TypeError: fetch failed
      4 |
      5 | export async function initializeTestEnv() {
    > 6 |   const testEnv = await initializeTestEnvironment({
        |                   ^
      7 |     projectId: "xxx",
      8 |     firestore: {
      9 |       rules: fs.readFileSync("firestore.rules", "utf8"),
      at async discoverEmulators (node_modules/@firebase/rules-unit-testing/src/impl/discovery.ts:31:15)
      at async initializeTestEnvironment (node_modules/@firebase/rules-unit-testing/src/initialize.ts:66:32)
      at async initializeTestEnv (lib/utils/unit-test.ts:6:19)
      at async Object.<anonymous> (lib/api-client/__tests__/user-report-test.node.ts:16:15)
    Cause:
    SocketError: other side closed

It seems await fetchImpl(makeUrl(hub, '/emulators')); in discoverEmulators throwed the SocketError: other side closed and the codes didn't catch it then the codes crashed, can we retry when discoverEmulators catches such an error?

Our Github workflow is similar to below:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: "20.15"

      - name: Install dependencies
        run: |
          npm install -g firebase-tools
          yarn

      - name: Lint
        run: yarn eslint .

      - name: Check TypeScript errors
        run: yarn tsc --noEmit

      - name: Run tests
        run: firebase emulators:exec --only "firestore,auth,storage" "jest --runInBand --updateSnapshot --detectOpenHandles --forceExit --coverage"

We invoke initializeTestEnv like below:

await initializeTestEnvironment({
    projectId: "xxx",
    firestore: {
      rules: fs.readFileSync("firestore.rules", "utf8"),
      host: 'localhost',
      port: 8080,
    },
    storage: {
      rules: fs.readFileSync("storage.rules", "utf8"),
      host: 'localhost',
      port: 9199,
    },
    hub: {
      host: 'localhost',
      port: 4400
    }
  })

We invoke it in each Jest unit test file in beforeAll, there are 5 Jest test files in our codebase, so this function is called 5 consecutive times.

Steps and code to reproduce issue

N/A, this is an occasional issue and only observed in GitHub Actions workflows.

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[tom-andersen]

Hi @jin-qin!

Retrying should be a safe approach as far as I know. It is possible that the emulator isn't ready to accept connections, so upon second attempt it will work. If the problem persists, please let us know.

[DellaBitta]

Yes, agreed! Generally this error means that the runtime couldn't connect to the emulator. If the problem persists, it would be helpful to know if the emulator has logged any errors when it was initialized.

[jin-qin]

Thank you folks for the quick responses!

The only error I got is SocketError: other side closed which caused by fetch API.

The weird part is, my 5 tests ran sequentially and the first one succeeded, but the second one failed by this issue, then the rest of the tests succeeded again, sometimes the order may be different.

My request is: is it possible for firebase-js-sdk to add a retry functionality while invoking fetch API to mitigate this kind of failure?

[DellaBitta]

I'll add the ability to configure retries to the Rules Unit Testing product backlog but it might take a while for the team to get to it since there's a viable workaround.

I've removed the Firestore label from this issue since it's the Emulator Hub that's closing the socket connection, not the Firestore emulator or product SDK.

It might be worth opening an issue in firebase-tools repo, which hosts the emulators, since their Emulator Hub is what's closing the socket connection. I went to create an issue on your behalf but their issue templates requires more details than I have here (your firebase-cli version, etc.). Please do link to this issue when you create one there so that they know that we sent you. :)

[jpreynat]

Just for traceability, we got the same issue. Happening only in CI on one test over hundreds of them, when moving to [email protected] and [email protected]. Really weird.

[jpreynat]

I opened an issue on the firebase-tools repo here: https://github.com/firebase/firebase-tools/issues/8383

However, I want to emphasize the fact that, in our case, it's moving from firebase@10 to firebase@11 that led to this random error.

Keeping the same version of firebase-tools, initially [email protected], or the latest [email protected], doesn't change anything, which points to an issue in the firebase-js-sdk repo where the new usage of the native Node fetch feature doesn't match the previous behavior with undici.

[singhaditya73]

Hi @DellaBitta, may I add a 3-retry loop around the fetch in discoverEmulators() to fix the occasional SocketError: other side closed in CI? No API change, just resiliency.