auth/internal-error calling `generateEmailVerificationLink` or `generatePasswordResetLink`

[umarhussain15]

Describe your environment

• Operating System version: Firebase Functions Environment
• Firebase SDK version: 7.0.0
• Library version: 7.0.0
• Firebase Product: auth

Describe the problem

I'm facing internal-error when calling the auth functions to generate Email verification link or password verification link. It throws the error randomly, some time it is successful, but its not consistent. There is no rate limit mentioned for the functions generateEmailVerificationLink and generatePasswordResetLink. Other operations of auth like getUser and setCustomClaims works fine in the same execution where the above mention two functions are failing. The complete error response is: {"error":{"code":400,"message":"TOO_MANY_ATTEMPTS_TRY_LATER","errors":[{"message":"TOO_MANY_ATTEMPTS_TRY_LATER","domain":"global","reason":"invalid"}]}}

Steps to reproduce:

The below code captures the flow of auth operation executions in my application.

Relevant Code:

let promises = [];
let auth = admin.auth();
let hrstart = process.hrtime();


async function testAuth(i) {
    let email = `email+test${i}@gmail.com`;
    let user;
    try {
        user = await auth.createUser({email, password: 'password1234567890'});
    } catch (e) {
        console.log('error creating user', email, e);
        // return Promise.reject(e);
    }

    try {
        await auth.setCustomUserClaims(user.uid, {abc: `testwqa`});
    } catch (e) {
        console.log('error writing user claims', email, e);
        // return Promise.reject(e);
    }

    try {
        user = await auth.getUser(user.uid);
    } catch (e) {
        console.log('error reading user ', email, user.uid, e);
        // return Promise.reject(e);
    }

    let link, plink;
    try {
        link = await auth.generateEmailVerificationLink(user.email, {url: `https://${projectId}.firebaseapp.com/redeem-reward`});
    } catch (e) {
        console.log('error generating email link for ', user.uid, e);
        // return Promise.reject(e);
    }

    try {
        plink = await auth.generatePasswordResetLink(user.email, {url: `https://${projectId}.firebaseapp.com/redeem-reward`});
    } catch (e) {
        console.log('error generating password link for ', user.uid, e);
        // return Promise.reject(e);
    }


    return {user, link, plink};
}

for (let i = 0; i < 3; i++) {
    promises.push(testAuth(i));
}

Promise.all(promises)
    .then(value => {
        let hrend = process.hrtime(hrstart);
        console.log(hrend);
        console.log(value)
    });

Error log:

[bojeil-google]

This is a also posted on stackoverflow: https://stackoverflow.com/questions/54703649/firebase-admin-sdk-auth-error-too-many-attempts-try-later

As suggested in the stackoverflow post, try to optimize your calls. There is no need to send a password reset and email verification link at the same time. It is also not a good practice (from user's perspective as they will get 2 emails). Password reset should verify the user's email making the verification email unnecessary.

Can you try making this change and report if you stop getting these errors?

[umarhussain15]

I have tried after removing the passwordResetLink call but the issue is still present. The calls are optimized now but still the error is present specifically on these calls.

On a side note I have looped auth.getUser 1000 iterations separately but there was no error in this operation.

[bojeil-google]

I can't tell where exactly you are getting throttled. You are making multiple write calls in parallel and then doing the same thing 3 times. You can definitely optimize further.

If you are batch creating users, then you should consider using importUsers API. This allows you to upload up to a 1000 users and it also supports setting custom attributes on creation. This will improve your performance and avoid getting rate limited.

[umarhussain15]

This is the scenario that my function will be executing in parallel, which I have simulated with Promise.all in my code. I'm able to create users and write their custom claims in parallel and there is no error, but I get the error when I try to generate the email link. I have updated my code to have each operation in a try catch to get exactly where the error is. As you can see in logs it fails only when creating email link.

[bojeil-google]

Ok, so you will end up getting throttled at some point as email link generation is not meant to be run in batches as you appear to be doing. You will need to apply some throttling on your end. If you can figure out the number of requests you are able to send before getting throttled, that could help determine whether throttling is excessive or not for this operation.

You are better off opening a support ticket for this. The support team will help connect you directly to the backend engineers in charge of this. The GitHub repo is not the right place for these issues. I can only help diagnose the issue and play middleman but can't look into the usage logs for your project or make changes to fix this.

[umarhussain15]

Ok. I will contact the firebase support with my issue. Thanks.

[umarhussain15]

There is a limit to the operation auth.generateEmailLink which is 20 QPS(queires per second)/ i.p address. The limit can be increased by presenting the use case to firebase for approval.

While the operation auth.generatePasswordLink has limits based on paln: quota of 150 emails/day for Spark plan, and 10,000 emails/day for Blaze plan.

Other operations limits are documented in firebase docs.

[hiranya911]

@bojeil-google should we add this info to our documentation?

[bojeil-google]

Ok, I will need to confirm these numbers first.

[batdevis]

Here's the link to the docs:

https://firebase.google.com/docs/auth/limits

[hiranya911]

@bojeil-google is there anything we need to do to resolve this?

[bojeil-google]

I wasn't able to verify the QPS limit per IP address. But I can lookup the email link generation limit per day for each operation and document it.

[lezsakdomi]

I experience the same error, but even on the first invocation.

Code: TL;DR operating on firestore and on RTDB, then generating a link on user creation

import * as functions from 'firebase-functions';
import * as admin from 'firebase-admin';
import * as sgMail from '@sendgrid/mail';

const useSg = Boolean(process.env.SENDGRID_API_KEY);

if (useSg) {
  sgMail.setApiKey(process.env.SENDGRID_API_KEY as string);
}

// noinspection JSUnusedGlobalSymbols
export const onUserCreate = functions.auth.user().onCreate(async (user) => {

  const userDoc = admin.firestore().collection('users').doc(user.uid);
  await userDoc.set({
    // ...
  });
  await userDoc.collection('...').doc("...").set({}); // just creating an empty document in Firestore

  const userRef = admin.database().ref('users').child(user.uid);
  await userRef.set({
    // ...
  });

  if (user.emailVerified) throw new Error("User email is already verified"); // impossible

  if (!user.email) throw new Error("User has no email address");

  if (useSg) {
    const link = await admin.auth().generateEmailVerificationLink(user.email); // causes error 400 for some reason

    await sgMail.send({
      to: {name: user.displayName, email: user.email},
      from: {name: "Planemo Team", email: "[email protected]"},
      subject: "Activate your registration",
      templateId: "...",
      dynamicTemplateData: {
        subject: "Activate your registration",
        name: user.displayName,
        link,
      },
    });
  } else {
    // ...
  }

});

Error shown in function logs:

Error: An internal error has occurred. Raw server response: "{"error":{"code":400,"message":"TOO_MANY_ATTEMPTS_TRY_LATER","errors":[{"message":"TOO_MANY_ATTEMPTS_TRY_LATER","domain":"global","reason":"invalid"}]}}"
    at FirebaseAuthError.FirebaseError [as constructor] (/srv/node_modules/firebase-admin/lib/utils/error.js:42:28)
    at FirebaseAuthError.PrefixedFirebaseError [as constructor] (/srv/node_modules/firebase-admin/lib/utils/error.js:88:28)
    at new FirebaseAuthError (/srv/node_modules/firebase-admin/lib/utils/error.js:147:16)
    at Function.FirebaseAuthError.fromServerError (/srv/node_modules/firebase-admin/lib/utils/error.js:186:16)
    at /srv/node_modules/firebase-admin/lib/auth/auth-api-request.js:1360:49
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7)

Tried with multiple projects, same result.

[bojeil-google]

It appears you are sending too many requests in a short period of time. I suggest you file a ticket with support and provide them details about your project.

[lezsakdomi]

No, I am sure that I am sending just one request. Checked the logs and the project is not even public. (ok, tried it once again but got the same result)

[vinay30]

Hey @bojeil-google , I'm having the same issue as @lezsakdomi

Calling once, getting code:400, message is TOO_MANY_ATTEMPTS_TRY_LATER, with domain: global and reason: invalid.

Any chance the IP could be completely blocked?

[vinay30]

Yup, it was an IP block. It's part of an E2E test we run, so we need to figure out a better way to run tests that also happen to touch firebase APIs.

[JakeMalis]

Bump?

{ Error: An internal error has occurred. Raw server response: "{"error":{"code":400,"message":"TOO_MANY_ATTEMPTS_TRY_LATER","errors":[{"message":"TOO_MANY_ATTEMPTS_TRY_LATER","domain":"global","reason":"invalid"}]}}" at FirebaseAuthError.FirebaseError [as constructor] (/workspace/node_modules/firebase-admin/lib/utils/error.js:43:28) at FirebaseAuthError.PrefixedFirebaseError [as constructor] (/workspace/node_modules/firebase-admin/lib/utils/error.js:89:28) at new FirebaseAuthError (/workspace/node_modules/firebase-admin/lib/utils/error.js:148:16) at Function.FirebaseAuthError.fromServerError (/workspace/node_modules/firebase-admin/lib/utils/error.js:187:16) at /workspace/node_modules/firebase-admin/lib/auth/auth-api-request.js:1490:49 at process._tickCallback (internal/process/next_tick.js:68:7) errorInfo: { code: 'auth/internal-error', message: 'An internal error has occurred. Raw server response: "{"error":{"code":400,"message":"TOO_MANY_ATTEMPTS_TRY_LATER","errors":[{"message":"TOO_MANY_ATTEMPTS_TRY_LATER","domain":"global","reason":"invalid"}]}}"' }, codePrefix: 'auth' }

[manwithsteelnerves]

Any workaround for avoiding this error? We have a test case where we check if the api is valid. This returns TOO_MANY_ATTEMPTS_TRY_LATER for a quick consecutive call.

[AndreuRosellOsuna]

I had the same issue when generateEmailVerificationLink was called in the backend after sendEmailVerification was invoked from the frontend, but it was really under 20 QPS and from 2 different IPs. Removing the call from the frontend fixed the problem.

[MihirSomani]

After spending 10+ hours trying to resolve this issue of getting rate limited on the 2nd query, I've had to completely remove Firebase Email Verification from my codebase. This just shows how limited platforms like Firebase are for production grade environments.

[timshirley]

One thing to ensure if you're running into this is that you're not using the same email address across multiple attempts. I found through experimentation that it will only allow a link to be generated once per minute per email address.. Which seems a perfectly reasonable thing to do to avoid spamming someone, but should still be mentioned in the limits documentation. I do not see this issue if I switch addresses between attempts.

[appsgenie]

I am also seeing this issue. I have not ran admin.auth().generateEmailVerificationLink in over 24hrs (from anywhere else or any user at all) and called it just now only one time (while deployed in the prod functions environment) and got this 400 TOO_MANY_ATTEMPTS_TRY_LATER error ... But, the client did also call the Firebase.auth.currentUser.sendEmailVerification() method around same time (obviously different IP). Could that be the issue?

[sudipstha08]

https://firebase.google.com/docs/auth/limits

How long does it takes for the user to get enable automatically, if it gets disabled ?

[ghost]

One thing to ensure if you're running into this is that you're not using the same email address across multiple attempts. I found through experimentation that it will only allow a link to be generated once per minute per email address.. Which seems a perfectly reasonable thing to do to avoid spamming someone, but should still be mentioned in the limits documentation. I do not see this issue if I switch addresses between attempts.

For anyone else looking at this where they're getting an error first time, this solved my problem. After several hours trying to work out what was going wrong, calls to sendEmailVerification involve creating a link too so count towards the ~1 call a minute per user limitation.

I'd been creating a link for an app feature, and also sending the link via a different channel at the same time (don't ask!). What this meant was that the first call to sendEmailVerification was effectively blocking the second being generated for 60s for the same email address/user.

[empeje]

If you don't handle that much user you can do a retry e.g.

exports.sendWelcomeEmail = functions.runWith({failurePolicy: true}).auth.user().onCreate(async (user) => {
    functions.logger.log("Running email...");
    const email = user.email;
    const displayName = user.displayName;
    const link = await auth.generateEmailVerificationLink(email, {
      url: 'https://mpj.io',
    });

    await sendWelcomeEmail(email, displayName, link);
});
``

[AnthonyNahas]

In my case I was calling this method like two times

one time after the user finishes registering in the frontend (angular)

user.sendEmailVerification() // in angular

`
``

and after that, I had a firebase cloud function that will be triggered whenever a user is created

```ts
export const onAuthUserCreated = functions.region('europe-west3')
  .auth.user()
  .onCreate(async (user: UserRecord) => {

// .. some code


const verificationURL = await getAuth()
          .generateEmailVerificationLink(user.email as string,

so removing the code in angular solved the problem!

I hope this could help someone <3