Fix auth emulator multi-tenant import/export

[aalej]

Description

Proposed fix for #5623

----- emulator export -----

Proposed fix is to export accounts into different json files. For example, a project that has the tenants tenant-1 and tenant-2 would generate:

1. account.json <- Contains accounts from default tenant
2. account-tenant-1.json <- Contains accounts from tenant-1 tenant
3. account-tenant-2.json <- Contains accounts from tenant-2 tenant

Process for emulator export

1. Get a list of tenants on the emulator using projects.tenants.list

const tenantsRes = await EmulatorRegistry.client(Emulators.AUTH).get<{
    tenants: Array<Tenant>;
}>(`/identitytoolkit.googleapis.com/v2/projects/${this.projectId}/tenants`, {
    headers: { Authorization: "Bearer owner" },
});
const tenants = tenantsRes.body.tenants.map((instance: Tenant) => instance.tenantId);

2. Loop through each tenant and create a file called account-${tenantId}.json

for (const tenantId of tenants) {
  const accountsFile = path.join(authExportPath, `accounts-${tenantId}.json`);
  logger.debug(
    `Exporting auth users in Project ${this.projectId} ${tenantId} tenant to ${accountsFile}`
  );
  await fetchToFile(
    {
      host,
      port,
      path: `/identitytoolkit.googleapis.com/v1/projects/${this.projectId}/accounts:batchGet?maxResults=-1&tenantId=${tenantId}`,
      headers: { Authorization: "Bearer owner" },
    },
    accountsFile
  );
}

Outuput export would look like

$ tree .
└── <emulator_export_path>
    └── auth_export
        ├── account.json
        ├── accounts-second-tenant.json
        ├── accounts-third-tenant.json
        └── accounts-${tenantId}.json

----- emulator import -----

Proposed fix is to import accounts from the different json files created from export.

Replace the endpoint being used from /identitytoolkit.googleapis.com/v1/projects/${this.projectId}/accounts:batchCreate(projects.accounts.batchCreate) to /identitytoolkit.googleapis.com/v1/projects/${this.projectId}/tenants/${tenantId}/accounts:batchCreate(projects.tenants.accounts.batchCreate). This will allow us to specify which tenant the account should belong to.

It looks like the the two APIs are almost the same.

1. projects.accounts.batchCreate
2. projects.tenants.accounts.batchCreate

Note: AFAICT when no tenantId is provided for the API projects.tenants.accounts.batchCreate, it will use the default tenant.

1. Making a POST request to a blank tenantId path should add the account to the default tenant, and return a instance of UploadAccountResponse
   • URL(actual project) - https://content-identitytoolkit.googleapis.com/v1/projects/<project_id>/tenants//accounts:batchCreate
   • URL(emulator) - https://content-identitytoolkit.googleapis.com/v1/projects/<project_id>/tenants//accounts:batchCreate
   • JSON body -

   {
     "kind": "identitytoolkit#DownloadAccountResponse",
     "users": [
       {
         "localId": "9GB64Wph3kXRkoSMZm3ZPWr01cPF",
         "createdAt": "1691019099015",
         "lastLoginAt": "1691019099016",
         "displayName": "Chicken Chicken",
         "providerUserInfo": [
           {
             "providerId": "google.com",
             "rawId": "2698285215994534916150568022884447626235",
             "displayName": "Chicken Chicken",
             "email": "[email protected]",
             "screenName": "chicken_chicken"
           }
         ],
         "validSince": "1691019147",
         "email": "[email protected]",
         "emailVerified": true,
         "disabled": false
       }
     ]
   }
   

Process for emulator import

1. Get a list of account json files in <emulator_export_path>/auth_export

const accountFiles = fs
  .readdirSync(authExportDir)
  .filter((fileName) => fileName.includes("accounts"));

2. Loop though each account file name and pass it through importFromFile

for (const accountFile of accountFiles) {
  const accountsPath = path.join(authExportDir, accountFile);
  const accountsStat = await stat(accountsPath);
  const tenantId = accountFile.replace(/accounts(-|)|.json/gm, "");
  if (accountsStat?.isFile()) {
    logger.logLabeled(
      "BULLET",
      "auth",
      `Importing accounts from ${accountsPath}`
    );

    await importFromFile(
      {
        method: "POST",
        host: utils.connectableHostname(host),
        port,
        path: `/identitytoolkit.googleapis.com/v1/projects/${projectId}/tenants/${tenantId}/accounts:batchCreate`,
        headers: {
          Authorization: "Bearer owner",
          "Content-Type": "application/json",
        },
      },
      accountsPath,
      // Ignore the error when there are no users. No action needed.
      { ignoreErrors: ["MISSING_USER_ACCOUNT"] }
    );
  } else {
    logger.logLabeled(
      "WARN",
      "auth",
      `Skipped importing accounts because ${accountsPath} does not exist.`
    );
  }
}

----- auth emulator batchGet endpoint -----

Noticed that the batchGet endpoint was not working as intended when passing the path parameter tenantId. A GET request to 127.0.0.1:9099/identitytoolkit.googleapis.com/v1/projects/${projectId}/accounts:batchGet?tenantId=${tenantId} will always return an object containing users from the default tenant.

Reference API: https://cloud.google.com/identity-platform/docs/reference/rest/v1/projects.accounts/batchGet

----- auth emulator only shows default tenants users in the UI -----

As mentioned here, "other tenants than the default using the Google Auth Provider are not being listed in the popup sign-in screen". Cause of this may be because the the link created is localhost:9099/emulator/auth/handler?<query_string>&tid=<tenant_id>, but the emulator is looking for req.query.tenantId. So the URL should be localhost:9099/emulator/auth/handler?<query_string>&tenantId=<tenant_id>. tenantId vs tid

Proposed solution is to get either req.query.tenantId or req.query.tid.

const tenantId = (req.query.tenantId || req.query.tid) as string | undefined;

Scenarios Tested

Using the sample web app in https://github.com/aalej/issues-5623.

Sample Commands

firebase emulators:start --export-on-exit=./users --import=./users --project demo-project

[mkurcius]

any news on this PR? I also have multi-tenant project and without these change, I cannot setup emulator state for local development

[jesdavpet]

Any way those of us in Firebase user-land can help to expedite this PR's review/readiness @firebase-ops?

Context: My team's doing a Google Cloud Identity Platform integration and we're unable to set up the emulator as needed for our local development environment until multi-tenancy is better supported.

Happy to help (if we can)!

[wdforson-sada]

any news on this PR? I also have multi-tenant project and without these change, I cannot setup emulator state for local development

I am embarrassed to admit that I just lost a full 2 days on trial-and-error-based debugging of broken import/export functionality in an inherited local dev setup that involves emulated Firebase auth with multi-tenancy...only to finally stumble upon #5623 and this open PR.

Given that this PR has been open for over a year now, would someone on this project at least be merciful enough to document this gap -- at a minimum in the online CLI reference, and ideally also in the help messages for the relevant CLI entrypoints?

[aalej]

Hey folks, apologies for the delay here. I'll try to get some time to work on this PR to resolve the merge conflicts and get this updated, as well as adding tests. I'll also ask someone from our team to review the changes.

[aalej]

Thanks for the review, CHANGELOG entry added in https://github.com/firebase/firebase-tools/pull/6217/commits/a4b9751610a7461fb05afdb90adfa8ca80830e48

[steveoh]

@joehan are we ready to move forward on this? Did the auth folks take a look yet?

[Dutch77]

It's really sad that a lot of devs are making hacky solutions while this fix lies here for almost two years now. :/

[AlexanderVorobyov]

Hi @joehan, any news on this?