Error: 2 UNKNOWN: Authentication backend unknown error

[michaelAtCoalesce]

trafficstars

Over the last 12 hours our firebase-admin node process has been experiencing the following error

Error: 2 UNKNOWN: Authentication backend unknown error

It appears to happen during a firestore transaction during the get, we do this in a few places and it seems to be happen transiently

[REQUIRED] Describe your environment

   "firebase-admin": "^11.0.1",
   node.js: "18.5.0"

between the periods of september 10th - 15th, we did not get this error. but we also got these issues on september 9th as well

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[google-oss-bot]

I found a few problems with this issue:

• I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.
• This issue does not seem to follow the issue template. Make sure you provide all the required information.

[hsubox76]

I am transferring this issue to the firebase-admin-node repo, as this is about the firebase-admin package.

[cherylEnkidu]

Hi,

Could you please file a support request so the team can have info about the your project?

[michaelAtCoalesce]

hey @cherylEnkidu i did - the firebase support case # is 10197247

[cherylEnkidu]

Hi @michaelAtCoalesce , are you still experiencing these errors recently?

[cherylEnkidu]

Close the issue since there is no updates. Please feel free to reopen it.

[samatcolumn]

We just experienced a massive burst of these out of nowhere:

[nicolas-ilr]

Hello, not a massive burst for us, but we saw the same problem @samatcolumn posted out of nowhere for some requests. I checked the Firestore health dashboard and didn't see any incident report. Anyone knows what can be causing this? Is it something to worry about?

[samatcolumn]

The issue seems to be quieting down now:

I don't think this is an SDK issue, but maybe the SDK team could get someone from the backend team to post an update here or on the status dashboard.

[hkchakladar]

Same here. Suddenly getting this error on pub/sub cron function.

Full log:

Error: 2 UNKNOWN: Authentication backend unknown error.
    at Object.callErrorFromStatus (/workspace/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client.js:330:49)
    at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)
    at process.nextTick (/workspace/node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78)
    at process._tickCallback (internal/process/next_tick.js:61:11)
Caused by: Error
    at Query._get (/workspace/node_modules/@google-cloud/firestore/build/src/reference.js:1450:23)
    at Query.get (/workspace/node_modules/@google-cloud/firestore/build/src/reference.js:1439:21)
    at exports.cron1hless5m.functions.pubsub.schedule.timeZone.onRun (/workspace/index.js:12987:4)

[MorenoMdz]

Yep same here.

[osdajigu]

We got the same error at the same time. It started from 22:25:15 (UTC).

[karkeeb]

we got them as well around the same time for around 10 minutes or so and then they disappeared. Any conclusion ?

[leandroz]

Getting them right now again, the same happened yesterday, a burst and then ok again.

[MorenoMdz]

Same here...

[samatcolumn]

Yep this is happening again!

[jlotito-scoir]

These errors are lining up with errors we are seeing.

[MorenoMdz]

@cherylEnkidu do you mind reopening this and or contacting support internally? I have a support ticket but it did not get any updates yet.

[samatcolumn]

cc @lahirumaramba

[karkeeb]

got this again. @cherylEnkidu / @lahirumaramba : can you please reopen / look at this ?

[inazalaei]

Did anyone find any solution? we are still having this issue.

[samatcolumn]

@cherylEnkidu we experienced another burst of these errors today from 6:40:25 - 6:41:00 (US Eastern Time). Can you please escalate this to Google Cloud? In my opinion these outages should count against Firestore's 99.999% multi-regional SLA which allows for only 5.26 minutes of downtime per year. We have exceeded that on this issue alone this year, and it is not the only kind of FIrestore downtime we have experienced.

[samatcolumn]

Also @cherylEnkidu one reason why this may be relevant to this repository: the previous response I got from Google Cloud is that it's the client's responsibility to retry with exponential backoff. I think that's nonsense (we can't know about every category of gRPC error!) but it may be something that the Admin Node SDK could handle internally.

[jbromberg]

Getting these errors randomly on different firestore operations (read and write).

[yilunc]

Seeing the same issue here! Was anybody able to contact support and receive a helpful response?

[samatcolumn]

Support gave me the extremely unhelpful response that this is within Firestore's SLA because it's my responsibility, as the consumer, to implement a retry policy. I strongly believe that retry should be done within the Firebase Admin SDK and not inside of every single app but 🤷

[oconnorjoseph]

This error is still occuring for us.

[ehedaya]

We're experiencing this, no idea what's causing it. Help.

[dschnare]

Likewise. We're noticing this occur periodically today for us.