Could not reach Cloud Firestore backend after upgrading to v9, but only in some projects

[johnnyoshika]

[REQUIRED] Describe your environment

• Operating System version: Windows 10
• Browser version: Chrome 97
• Firebase SDK version: 9.6.5
• Firebase Product: Firestore

[REQUIRED] Describe the problem

After upgrading to Firebase v9 from v8, I'm encountering the following error in 2 of 4 environments:

@firebase/firestore: Firestore (9.6.5): Could not reach Cloud Firestore backend. Backend didn't respond within 10 seconds.
This typically indicates that your device does not have a healthy Internet connection at the moment. The client will operate in offline mode until it is able to successfully connect to the backend.

Here are the environments and results:

GCP Project ID	GCP Firestore Region	After upgrading to v9
Firebase Emulator	None (local)	Works fine
{project}-dev	us-central	Works fine
{project}-prod	us-central	Could not reach Cloud Firestore backend
{project}-prod-ca	northamerica-northeast1	Could not reach Cloud Firestore backend

The code in all 4 environments are identical and the CI/CD pipeline for the 3 GCP projects are identical. It's particularly noteworthy that {project}-dev works fine while {project}-prod and {project}-prod-ca don't.

Steps to reproduce:

To see the results, I created these 2 test pages. They both use SDK v9:

{project}-dev: https://student-dev.examind.io/firestore {project}-prod-ca: https://student-ca.examind.io/firestore

You will notice that {project}-dev loads firestore data just fine, while {project}-prod-ca doesn't and outputs this to the console after 10 seconds:

There are a couple of related issues (https://github.com/firebase/firebase-js-sdk/issues/5667, https://github.com/firebase/firebase-js-sdk/issues/5932), but seeing different results in different projects with the same codebase may be a unique opportunity to troubleshoot this.

Relevant Code:

const query = collection(db, 'path-to-collection');
onSnapshot(query, snap => {
  console.log('Do something');
});

[dconeybe]

@johnnyoshika Thank you for the detailed report. This is indeed odd behavior. Would you by any chance be able to share the source code for those web pages so that I could reproduce myself? Also, would you be able to modify those pages to enable Firestore debug logging?

Enable debug logging by calling setLogLevel:

firebase.firestore.setLogLevel('debug');

In the meantime, I'll look into what I can from my end.

[johnnyoshika]

@dconeybe Thanks for the follow up. I've enabled debug logging. I've also created simpler test pages so that I can share the source code.

New test pages:

https://student-dev.examind.io/firestore5 (works fine) https://student-ca.examind.io/firestore5 (Could not reach Cloud Firestore backend)

Source (React Component):

import { db } from '@foo/web/dist/config/firebase'; // Configured firestore instance that comes from a shared monorepo package
import { doc, onSnapshot, setLogLevel } from 'firebase/firestore';
import { useEffect, useState } from 'react';

setLogLevel('debug');

const Firestore5 = () => {
  const [document, setDocument] = useState<any>(undefined);

  useEffect(() => {
    onSnapshot(doc(db, 'widgets', 'one'), snap => {
      if (snap.exists()) setDocument(snap.data());
    });
  }, [setDocument]);

  if (!document) return null;

  return <div>{document.name}</div>;
};

export default Firestore5;

I also created a version where a new Firebase app is instantiated with only Firestore enabled. The test site is here but the results are the same:

https://student-dev.examind.io/firestore6 (works fine) https://student-ca.examind.io/firestore6 (Could not reach Cloud Firestore backend)

Source (React Component):

import { initializeApp } from 'firebase/app';
import {
  doc,
  getFirestore,
  onSnapshot,
  setLogLevel,
} from 'firebase/firestore';
import { useEffect, useState } from 'react';

const app = initializeApp({
  apiKey: 'redacted',
  authDomain: 'redacted',
  projectId: 'redacted',
  storageBucket: 'redacted',
  messagingSenderId: 'redacted',
  appId: 'redacted',
});

const db = getFirestore(app);

setLogLevel('debug');

const Firestore6 = () => {
  const [document, setDocument] = useState<any>(undefined);

  useEffect(() => {
    onSnapshot(doc(db, 'widgets', 'one'), snap => {
      if (snap.exists()) setDocument(snap.data());
    });
  }, [setDocument]);

  if (!document) return null;

  return <div>{document.name}</div>;
};

export default Firestore6;

[dconeybe]

Very interesting. I can easily reproduce with those links that you sent. Nothing obvious jumps out to me from the debug logs, which isn't too surprising since the "Could not reach Cloud Firestore backend" errors are pretty generic. Are you by chance able to generate the HTML and JavaScript with minification and obfuscation disabled? That way I could set breakpoints and step through some code.

[johnnyoshika]

@dconeybe Yes, I'll re-build with sourcemaps included. I'll let you know when it's ready.

[johnnyoshika]

@dconeybe Updated with source maps. The firestore6 pages will be easiest to debug as they don't rely on a separate package in the monorepo.

Looking for the Firestore6/index.tsx file in dev tools will allow you to put breakpoints in the code:

[schmidt-sebastian]

Debugged this @ehsannas:

Current theory is that this has to do with AppCheck enforcement. In https://student-ca.examind.io/firestore5 we see that we open a connection (connection 1), then we get a new AppCheck token and open connection 2. We keep getting responses on connection 1 but ignore those since connection 2 is the connection we are now listening on. So there seems to be some problem with the AppCheck token refresh that we have to look into.

Does the -dev project use AppCheck? Do you see the same behavior with v9.6.4? We changed the AppCheck handling logic with 9.6.5

[johnnyoshika]

@schmidt-sebastian The 2 environments -dev and -ca should be identical, so AppCheck is enabled in -dev. Here are the AppCheck settings for both projects:

The codebase is also identical, but this morning an automatic deployment was triggered for -dev that wiped out the firestore tests pages there. They should now be fixed and you can access the test -dev pages again.

I just tried v9.6.4 in -ca, but I'm getting a strange Component app-check has not been registered yet error after deploying there, so I reverted back to v9.6.5.

Test pages:

https://student-dev.examind.io/firestore5 (works fine) https://student-ca.examind.io/firestore5 (Could not reach Cloud Firestore backend)

https://student-dev.examind.io/firestore6 (works fine) https://student-ca.examind.io/firestore6 (Could not reach Cloud Firestore backend)

[johnnyoshika]

@schmidt-sebastian You may find this one other piece of information interesting. Here is another test page:

https://moderator-ca.examind.io/firestore

It uses the same Firestore backend as https://student-ca.examind.io/firestore at northamerica-northeast1 and it works fine. It even uses the same monorepo package to configure the firestore instance (e.g. const db = getFirestore(app)). The only difference is that indexed DB persistence is not enabled in moderator-ca.examind.io while it is in student-ca.examind.io.

Note however, that indexed DB persistence IS enabled on student-dev.examind.io, so it doesn't look like the problem is isolated to offline persistence.

[johnnyoshika]

@schmidt-sebastian The most bizarre thing just happened. It was becoming a pain to run the deployment through my CI/CD pipeline with every change I'm experimenting with, so I deployed the -ca site manually with the Firebase CLI as follows:

firebase deploy --only hosting

Surprisingly Firestore worked when I deployed that way. Then I rolled back to a previous deployment that went through my CI/CD pipeline and once again I was faced with Could not reach Cloud Firestore backend.

The CI/CD pipeline I'm using is GitHub Actions and this is the action that I'm using: https://github.com/FirebaseExtended/action-hosting-deploy

Everything is hosted at Firebase, so I'm not sure if you're able to access these deployment artifacts or not:

Note again that deploying the site to -dev using Github Actions doesn't pose any problems, so this is all still a mystery.

[johnnyoshika]

@schmidt-sebastian I'd be happy to hop in a meeting and screen share if that'll be helpful. I've been trying all kinds of different permutations to try and figure out what's happening, but I'm completely stumped. If I can't resolve this soon I fear that I'll have to take the painful steps to revert back to version 8 of the SDK.

Other things I've tried:

• Firebase Web SDK v9.5.0 (before AppCheck was introduced for Cloud Firestore) -> still the same problem
• Disabling AppCheck (i.e. removing this code: initializeAppCheck(app, ...) -> still the same problem

My observation is still the same: 2 almost identical projects (-dev and -ca), yet one exhibits problems while the other doesn't.

The only difference between -dev and -ca is the Firestore region. Note: we have another project with Firestore in the same region as -dev that exhibit the same behavior as -ca but I don't have the luxury to experiment with that one because we have real users there.

[johnnyoshika]

@schmidt-sebastian and team, unfortunately I had to revert my projects back to Firebase Web SDK v8. I'd still like to help you solve this problem, so all of my offers above stand. I'd also be happy to redeploy v9 into these projects for short periods at a time. I'll be watching this thread for any messages. Thanks for your help!

[schmidt-sebastian]

We will be fixing the AppCheck issue shortly. If the connection issues still persist afterwards, we can take another look at your project. Right now, we would like to reduce the noise on this reproduction and remove the additional requests issued by AppCheck.

[johnnyoshika]

@schmidt-sebastian that sounds very promising 👍

[johnnyoshika]

@schmidt-sebastian Any update on this? I'm keen to get my apps upgraded to Web SDK v9.

[jlanche]

I would like to add that this also appears to be an issue with local connections using the emulators. I'm using AngularFire but the logs are identical to @johnnyoshika 's. I'll paste below. Frankly, I hadn't used the emulators in a while so I can't say for sure what version specifically started causing this in my setup but my current version of the sdk 9.6.7 connects to production resources just fine. It's only when using the emulators that it fails.

[johnnyoshika]

@jlanche Very strange. My experience is the opposite. My apps connect to the emulator just fine and also to one of the 3 production projects, but fails on 2 of the production projects.

[jlanche]

@jlanche Very strange. My experience is the opposite. My apps connect to the emulator just fine and also to one of the 3 production projects, but fails on 2 of the production projects.

It is odd. The only common denominator in our setups is the sdk. Unfortunately I can't downgrade at this point without downgrading both AngularFire and Angular itself which had some migration work involved. So I'm in a bit of a bind. I also tried several earlier v9 versions of the sdk but no luck.

[johnnyoshika]

@jlanche You may be running into a problem I run into often with Windows 10 and Chrome. I didn't mention this problem in this issue because it's been happening since SDK v8. Often the connector with Firestore's Emulator times out using Chrome-like browsers, which includes Edge. Have you tried using Firefox? It's much more reliable for me when using the Emulator.

Note that this doesn't seem to be a problem when using a Mac.

[schmidt-sebastian]

The AppCheck issue has been resolved. If you still have problems, please share your debug logs. It would also help if you try with experimentalForceLongPolling (https://firebase.google.com/docs/reference/js/v8/firebase.firestore.Settings#experimentalforcelongpolling)

[johnnyoshika]

@schmidt-sebastian Great news, I'll try it soon. In what version of the SDK was it resolved?

[ehsannas]

The App Check fix should be in v3.4.5

[patrickk]

Using firebase v9.6.7 and still getting this error intermittently - wasn't v3.4.5 included in v9.6.7 @ehsannas @schmidt-sebastian?

[davidmyersdev]

Still running into this in v9.6.8.

[schmidt-sebastian]

Yes, it was released in 9.6.7: https://firebase.google.com/support/release-notes/js

[davidmyersdev]

@schmidt-sebastian I am still running into this error as of v9.6.9 in my local emulator. Things are working fine when I connect directly to my remote Firestore project though.

[juane1000]

@schmidt-sebastian I am still running into this error as of v9.6.9 in my local emulator. Things are working fine when I connect directly to my remote Firestore project though.

Unfortunately, I am too. Latest firebase sdk "9.6.10". Works fine in production but emulators give me the warning. I am also getting the "FirebaseError: Failed to get document because the client is offline." error when trying to read data. Writing seems to just not throw an error but just hangs while logging those warnings shown by the OP.

I've also tried running the application (web app) in Firefox instead of Chromium based browser as johnnyoshika recommended. I also happen to be on Windows 11. Not sure if that is a factor.

(Edit) It's worth mentioning that I'm using the compatibility library for @angular/fire . I'm not sure if that is contributing to the issue or not being affected by the fixes to the sdk.

[ehsannas]

@juane1000 @voraciousdev @patrickk

I'm not able to reproduce this issue locally (although I'm on a Mac and not using AngularFire). I'm trying to better understand if there's a common usage pattern here.

• Are you all experiencing "Could not reach Cloud Firestore backend" error when using the emulator?
• Are you using Windows?
• Are you using AngularFire?
• Are you using SDK 9.6.9 or higher?
• Would you be able to provide a minimal repository that we can clone to reproduce the issue?

[ehsannas]

@johnnyoshika I know you mentioned that you have experienced this error in production. Is that still the case with the latest SDK (9.6.10)?

[davidmyersdev]

@ehsannas, I'm only experiencing this issue with the emulator.

• I'm on MacOS, but I'm running the emulator in a docker container (using this build).
• I'm using the JS SDK directly at v9.6.10.
• I don't have a reproducible repo together, but I will try to get one set up.

Here is the exact error I'm seeing.

@firebase/firestore: Firestore (9.6.10): Could not reach Cloud Firestore backend. Backend didn't respond within 10 seconds.
This typically indicates that your device does not have a healthy Internet connection at the moment. The client will operate in offline mode until it is able to successfully connect to the backend.

[juane1000]

@ehsannas

I have done some isolated testing using the cdn sdk v9.6.10 with no framework and I can connect to the emulators. Reading data was no problem as well. This was also behind authenticated requirements.

Then I created a simple Angular project and used the vanilla sdk (9.6.10) via npm and this also worked correctly when reading data.

Regarding my original setup, the one with the issue of the OP, I realized that the imports from AngularFire that take the configuration json objects for the individual firebase services were not changed to use the compatibility layer. And as I'm using the compatibility layer to maintain the old API, this caused a mismatch. Long Story short, it got rid of the original error of this thread. Unfortunately it lead to a permissions error that prevented the data from being retrieved. I suspect this is part of the ongoing issues the AngularFire library is working through since v9 sdk switch.

That being said, my remaining permissions error likely falls out of purview of this thread. I will go to the AngularFire repo to report this further. All seems in order with the latest sdk on my end 👍

EDIT: All my testing is on a Windows 11 machine