firebase-tools icon indicating copy to clipboard operation
firebase-tools copied to clipboard

RESOURCE_EXHAUSTED error for realtime listeners on firestore emulator

Open FloKimmel opened this issue 6 months ago • 0 comments

[REQUIRED] Environment info

firebase-tools: 14.4.0 Platform: macOS 15.5 NodeJS: 22.11.0 Firebase JS SDK: 11.8.0 Vitest: 3.0.6

[REQUIRED] Test case

I've written a firestore wrapper in typescript. While unit testing it, I've experienced the error described below. I've replaced all custom wrapper code with the native firebase js sdk code leaving me with this:

import { afterEach, assert, describe, test } from 'vitest';
import { doc, onSnapshot, query, setDoc, where, collection, getFirestore, connectFirestoreEmulator } from 'firebase/firestore';
import { initializeApp } from 'firebase/app';
import { randomUUID } from 'node:crypto'; 

const host = '127.0.0.1';
const port = 8080;
const projectId = 'demo-debug';
const firebaseApp = initializeApp({ projectId });
const firestore = getFirestore(firebaseApp);
connectFirestoreEmulator(firestore, host, port);

afterEach(() =>
  fetch(`http://${host}:${port}/emulator/v1/projects/${projectId}/databases/(default)/documents`, {
    method: 'DELETE',
  }),
);

// Note the { repeats: 1 } as the second argument of `describe`
describe('debug RESOURCE_EXHAUSTED error', { repeats: 1 }, () => {
  test('listener should receive updates', async () => {
    // Arrange
    const collectionName = randomUUID().toString();
    const existingDocs = await Promise.all(
      Array.from({ length: 5 }).map(async (_, index) => {
        const id = `doc${index}`;
        const obj = { prop: index };
        await setDoc(doc(firestore, collectionName, id), obj);
        return obj;
      }),
    );

    const objToAdd = { prop: existingDocs.length };
    const expectedInitially = [ ...existingDocs ];
    const expectedAfterUpdate = [...expectedInitially, objToAdd];

    const result = new Promise<object[][]>((resolve) => {
      const data: object[][] = [];
      const unlisten = onSnapshot(collection(firestore, collectionName), (snapshot) => {
        data.push(snapshot.docs.map((doc) => doc.data()));
        if (data.length === 2) {
          unlisten();
          resolve(data);
        }
      });
    });

    // Act
    await setDoc(doc(firestore, collectionName, `doc${existingDocs.length}`), objToAdd);

    // Assert
    const [initially, afterUpdate] = await result;
    assert.deepEqual(initially, expectedInitially);
    assert.deepEqual(afterUpdate, expectedAfterUpdate);
  });
});

[REQUIRED] Steps to reproduce

Run npx firebase emulators:exec 'vitest run ./path/to/testfile/debug.test.ts'

[REQUIRED] Expected behavior

  • Test(s) run to completion without showing errors in the console.
  • Test execution does not slow down for potentially additional tests coming after the one shown above.

[REQUIRED] Actual behavior

Seeing the error below on the console. In addition if there were more tests in the test suite interacting with firestore (emulator) coming after the test shown above, these tests would get extremely slow and eventually fail due to test timeout. Note that this problem does not appear every time. I've experienced it for about 50% of the test runs.

[2025-05-25T10:49:36.745Z]  @firebase/firestore: Firestore (11.8.0): GrpcConnection RPC 'Listen' stream 0x6ad035e3 error. Code: 8 Message: 8 RESOURCE_EXHAUSTED: Received message larger than max (3236678593 vs 4194304)

stderr | src/debug.test.ts > debug RESOURCE_EXHAUSTED error > listener should receive updates
[2025-05-25T10:49:36.746Z]  @firebase/firestore: Firestore (11.8.0): FirebaseError: [code=resource-exhausted]: 8 RESOURCE_EXHAUSTED: Received message larger than max (3236678593 vs 4194304)
[2025-05-25T10:49:36.746Z]  @firebase/firestore: Firestore (11.8.0): Using maximum backoff delay to prevent overloading the backend.

Additional information & findings:

  • The problem disappeared after using the real firestore instance instead of the emulator. That's why I think this is not a problem of the firebase js sdk
  • The problem disappeared after setting { repeats: 0 } as the second argument for describe. At least I did not see the error after running it a hundred times. This indicates that listening and unlistening in quick succession is required for the error to appear
  • Increasing the number of documents existing before attaching the listener increases the chance of the error to appear
  • The Received message size shown in the error message can vary greatly from about ~500MB to ~3GB (without changing the code ofc.)
  • I have no firestore pre-test-run data seeding via the --import CLI argument. This actually makes it much more surprising how on earth to get to the ~3GB (!) mentioned in the error logs while only inserting/reading 6 very small documents (or rather 12 because of the additional execution due to the repeat).

firebase-debug.log firestore-debug.log

Thank you very much in advance for your time and effort!

FloKimmel avatar May 25 '25 13:05 FloKimmel