"Unhandled error cleaning up build images." - deploying from same GCP project

[akauppi]

[REQUIRED] Environment info

• firebase-tools: 11.3.0
• Platform: Cloud Build; same GCP project as the Firebase project

[REQUIRED] Test case

• deploy a Firebase project with Cloud Functions, "onto itself", i.e. using the same GCP project that Firebase has automatically created.

[REQUIRED] Steps to reproduce

skip

[REQUIRED] Expected behavior

Deployment works, since the project has enough access rights.

Firebase would clean up Cloud Function deployment cache, as it aims to do.

[REQUIRED] Actual behavior

Deployment fails.

The cleanup fails.

Here's an excerpt from the log:

Step #6: === Deploying to 'groundlevel-jun-22'...
Step #6: 
Step #6: i  deploying firestore, functions
...
Step #6: i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
Step #6: i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
Step #6: i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
Step #6: ✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
Step #6: ✔  functions: required API cloudbuild.googleapis.com is enabled
Step #6: ✔  functions: required API cloudfunctions.googleapis.com is enabled
Step #6: i  functions: preparing codebase default for deployment
Step #6: i  functions: preparing ./functions directory for uploading...
Step #6: i  functions: packaged /workspace/packages/backend/functions (53.92 KB) for uploading
Step #6: i  firestore: latest version of ./firestore.rules already up to date, skipping upload...
Step #6: ✔  firestore: deployed indexes in ./firestore.indexes.json successfully
Step #6: ✔  functions: ./functions folder uploaded successfully
Step #6: ✔  firestore: released rules ./firestore.rules to cloud.firestore
Step #6: i  functions: updating Node.js 16 function userInfoShadow_2(europe-west2)...
Step #6: ⚠  functions: failed to update function projects/groundlevel-jun-22/locations/europe-west2/functions/userInfoShadow_2
Step #6: Failed to update function projects/groundlevel-jun-22/locations/europe-west2/functions/userInfoShadow_2
Step #6: 
Step #6: Functions deploy had errors with the following functions:
Step #6: 	userInfoShadow_2(europe-west2)
Step #6: i  functions: cleaning up build files...
Step #6: ⚠  functions: Unhandled error cleaning up build images. This could result in a small monthly bill if not corrected. You can attempt to delete these images by redeploying or you can delete them manually at https://console.cloud.google.com/artifacts/docker/groundlevel-jun-22/europe-west2/gcf-artifacts
Step #6: 
Step #6: Error: There was an error deploying functions
Finished Step #6
ERROR
ERROR: build step 6 "us-central1-docker.pkg.dev/ci-builder/builders/firebase-emulators:11.3.0" failed: step exited with non-zero status: 2

The way I read that is that firebase-tools first checks that there are enough access rights:

Step #6: i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
Step #6: ✔  artifactregistry: required API artifactregistry.googleapis.com is enabled

..but it then tries to do something that's beyond those rights (delete the earlier cached images).

Screenshot of the cached images

Attempting work-around

I removed the cached images manually, and retried the deployment.

It still fails.

I removed the user_info_shadow2__ folder, leaving an empty gcf-artifacts. Retried.

It still fails in the same way.

Since there were some artifacts there from earlier runs (7 and 12 days ago), either this is something in 11.3.0 (I upgraded in the mean time), or I may have inadvertently changed some access rights.

More info

Manual deployment works, with firebase-tools 11.3.0. That is, my access rights (as the owner of the GCP project) are enough.

This also used to work, like months back, but I was using Container Registry at the time. I have not done CI deployments since moving to Artifact Registry.

---

Screenshot of the gcd-artifacts folder

Note that the deployment region is europe-west2, a bit unusual.

[akauppi]

Corrections:

It doesn't have to do with firebase-tools version.

I needed to enable Service Account User:

Since this is not mentioned in the Firebase docs (I believe), I'll leave this issue open. Let me know if you wish me to edit it.

Note: The inability to clean up was separate from the actual error that fails the deployment. Can do two tickets if you think both of these are worth doing something about.

[khenneuse]

Thanks @akauppi I created a pull request to update the documentation in the community build repo

There should be an update to the firebase tool to print a warning about the service account not having access when that happens. It would be a clearer signal as to the issue.

The build did pass when run as a single step

  - name: gcr.io/<PROJECT_ID>/firebase
  args: ["deploy", "--project=<PROJECT_ID>"]

But failed when run as individual steps

  - name: gcr.io/<PROJECT_ID>/firebase
    args:
      ["deploy", "--project=<PROJECT_ID>", "--only=functions", "--force"]
  - name: gcr.io/<PROJECT_ID>/firebase
    args: ["deploy", "--project=<PROJECT_ID>", "--only=hosting"]
  - name: gcr.io/<PROJECT_ID>/firebase
    args: ["deploy", "--project=<PROJECT_ID>", "--only=firestore"]
  - name: gcr.io/<PROJECT_ID>/firebase
    args: ["deploy", "--project=<PROJECT_ID>", "--only=storage"]

This may be an issue inside of the package that it is not capture or checking the exit code of each step. Therefore only the exit code of the last step in the chain is reported to Cloud Build.

[zouhirdev]

@akauppi hello how can i fix this please im new to firebase im having the same error i cant find the service account permissions to edit them please help

[joehan]

Hey @zouhirdev - you should be able to see and edit these permissions here: https://console.cloud.google.com/cloud-build/settings/service-account

If you are unable to see that page, you might not have the necessary permissions on your project, and should ask a project owner to help you out.