friendlyeats-web icon indicating copy to clipboard operation
friendlyeats-web copied to clipboard

nextjs-end rollout failed

Open v0lsh3bnik opened this issue 1 year ago • 10 comments

I am using the nextjs-end repo for firebase app hosting, the build is working fine but i get an error on the rollout. I looked through the logs but nothing. Can someone help me please ?

v0lsh3bnik avatar Jul 29 '24 01:07 v0lsh3bnik

I currently have the same issue, I am following the tutorial step by step and App Hosting is failing image

WCressman avatar Aug 01 '24 02:08 WCressman

In my case , I found out the issue by looking in Google Cloud > Logging > Logs explorer > SEVERITY > Error :

google.cloud.run.v2.Services.CreateService returns this error :

template.containers.resources.limits.cpu: Invalid value specified for cpu.
For the specified value, maxScale may not exceed 10.
...

Solved by adding the following lines to apphosting.yaml :

# Settings for Cloud Run
runConfig:
  minInstances: 0
  maxInstances: 10
  concurrency: 50
  cpu: 1
  memoryMiB: 512

I guess the Rollout Failure should be more verbose than that.

ni7r0x avatar Aug 02 '24 13:08 ni7r0x

I had the same issue with my nexjs deployments: Google Cloud Build was successfully building my app, but in firebase console, my deployment launch was failing. I've added this configuration in my apphosting.yaml that @ni7r0x commented (tks) and my deployment was completed

It would be great if the official docs mentioned this apphosting.yaml configuration file in the Basic Deployment section, and tell us if this file is required.

image image

AntonioGally avatar Sep 12 '24 01:09 AntonioGally

Had the same mildly frustrating issue. Thank you @ni7r0x.

gyula-szucs avatar Oct 12 '24 22:10 gyula-szucs

Mine just says all of a sudden: Ready condition status changed to False for Revision backlink-space-build-2024-10-28-003 with message: Deadline exceeded

r-dev-limited avatar Oct 28 '24 10:10 r-dev-limited

i also have the same issue now, i have added the apphosting.yaml in my project but still i get the same error, one more thing is that at the start i was able to deploy 2 build to apphosting but after working on my app for a bit i started to get this error.

Screenshot 2024-11-19 at 6 37 06 PM

RebinX1 avatar Nov 19 '24 15:11 RebinX1

Currently experiencing the same issue with a valid apphosting.yaml file present in my project. Google console shows in the logs that the build was successful in the cloud, and my build locally is fine, but the rollout fails every time.

ItsTeccs avatar Jan 28 '25 05:01 ItsTeccs

I'm suddenly getting failed deploys with a successful build as well, just like in @RebinX1 screenshot.

Had to delete and redeploy the backend to get a working deploy. Still seeing "Resource 'my-backend' already exists." errors in the GCP logs on re-deploy.

vojdan avatar Jan 28 '25 08:01 vojdan

Having the same issue, it looks like next build is not being run, and the official docs don't explain how to get the environment to build next-js apps for some reason, It thinks it's an angular application, which is not the case. Please, the Firebase team, make this very clear in the documentation.

josephkabba avatar Feb 04 '25 09:02 josephkabba

I have the same problem too. it looks like the build was completed, and the revisions in Cloud Run show that the build was successful, but firebase still shows that the build have failed in the console.

ItsBenDov avatar Apr 17 '25 09:04 ItsBenDov

same problem, and i fixed apphosting.yaml config problem, the revisions in Cloud Run show that the build was successful, no any error. but firebase still shows that the build have failed in the console.

Msekoo avatar Apr 18 '25 08:04 Msekoo

Looks like that's happening for a lot of users see this reddit thread: https://www.reddit.com/r/Firebase/comments/1k1f321/error_occurred_in_app_hosting_rollout/

report it to the firebase team for them to fix it.

ItsBenDov avatar Apr 18 '25 09:04 ItsBenDov

Same here. The build was successful but the rollout failed. And the logs explorer says: "The request was invalid: build "build-2025-04-18-002" was not found and is invalid"

Image

jimmy-chiang avatar Apr 18 '25 12:04 jimmy-chiang

This is their response:

"The App Hosting expert team has now identified the cause. They found that there’s a brief period of time between when you create a build and when it’s safe to start the rollout. In the cases they analyzed, the build and the rollout were triggered in the same second most likely through a script. As a temporary workaround, they recommend waiting a couple of seconds between triggering the build and the rollout. The team is already working on a fix to handle this better in the future."

But I simply don't understand what I should do? like how can I make it wait a couple of seconds?

ItsBenDov avatar Apr 19 '25 18:04 ItsBenDov

I'm back to this thread because I was able to get successful build(s) by deleting the container in cloud run then re-building. That was, until last night & this morning. No matter what my local build is good, my cloud build is good, but my rollout fails. This is a critical level issue in my own opinion given the amount of users experiencing problems and the fact that the platform is wholly unusable if we cannot get a successful rollout or meaningful logs to tell us why we are not getting successful rollouts. I've lost two weekends of development time in the last few months because of this issue.

ItsTeccs avatar Apr 19 '25 20:04 ItsTeccs

@ItsBenDov When you receive a response from Firebase in context of what to do, could you please post here again as well?

What category did you file this under when speaking to Firebase support? I will likely submit a ticket too but it would make sense for those of us experiencing issues to attempt to categorically pool the support tickets to make it clearer to Firebase that this is a serious issue affecting many users.

ItsTeccs avatar Apr 19 '25 21:04 ItsTeccs

I reported it here, and marked the urgency as: "This is preventing me from using Firebase. There is no workaround." under "App hosting" and "NodeJS". The support team responded to the ticket quickly, but I think that as soon as more people will report this very critical issue, they will try to address it asap.

ItsBenDov avatar Apr 19 '25 21:04 ItsBenDov

read the document and tried following config in apphosting.yaml, but got no luck 😞

# try workaround to escape the rollout problem from Firebase.
scripts:
  buildCommand: next build --no-lint && sleep 5
  runCommand: node dist/index.js && sleep 5

rzfang avatar Apr 21 '25 06:04 rzfang

Here’s how I resolved this issue:

Solution 1: Delete and recreate the Firebase App Hosting backend

The simplest fix in many cases is to delete your existing Firebase App Hosting backend and create a new one from scratch. This worked reliably across multiple projects for me.

Solution 2: Manually edit the Cloud Build history 1. Go to Google Cloud Console > Cloud Build > Build History. 2. Click on the YAML tab for a recent build. 3. Edit the file and remove any entries that reference build dates not associated with your listed builds. 4. Go back to Firebase, and manually trigger a rollout of your most recent deployment to GitHub.

Note: This approach is not recommended, and you’ll have to repeat it every time you deploy. I only use it on one of my apps where downtime isn’t an option.

Important note for users with GoDaddy DNS:

If you’re using GoDaddy’s nameservers, be cautious. After deleting and recreating a Firebase backend, I experienced issues where Google Cloud was effectively blocked by GoDaddy’s internal systems. It took multiple escalations for their support to acknowledge the issue. I strongly recommend switching to Google Cloud DNS if you plan to delete and recreate your Firebase project that had a custom domain to avoid such delays.

How to avoid this issue in the first place:

In my experience, this problem often arises when you push to GitHub and deploy back-to-back too quickly—like realizing you missed something immediately after deploying and pushing a second fix right away. Avoid making consecutive rapid deployments. Let the first deployment fully complete before initiating another.

This is not helpful, you may have another problem

Msekoo avatar Apr 21 '25 13:04 Msekoo

According to Firebase support, it seems like it is happening when we attempt to rollout shortly after creating a backend. They're currently investigating to solve this issue.

julian-printemps avatar Apr 22 '25 02:04 julian-printemps

According to Firebase support, it seems like it is happening when we attempt to rollout shortly after creating a backend. They're currently investigating to solve this issue.

Image

I created a new backend to test this theory out and waited about 10 minutes before deploying for the first time; I disabled auto-rollout and clicked "finish" rather than "finish and deploy" to let the backend sit idle before deploying initially.

This is the same project, same branch, same commit, same configuration on two different backends. One fails but the other succeeds.

ItsTeccs avatar Apr 22 '25 03:04 ItsTeccs

According to Firebase support, it seems like it is happening when we attempt to rollout shortly after creating a backend. They're currently investigating to solve this issue.

My backend has been created and used for a long time, but I have not been able to publish it successfully since recently.

Msekoo avatar Apr 22 '25 03:04 Msekoo

An incident has been published on the Firebase Status Dashboard.

https://status.firebase.google.com/incidents/HkK8snnXw4jYrUVfYbNw

keiziiisan avatar Apr 23 '25 12:04 keiziiisan

The same, also creating a new backend doesn't solve the problem for me. It's crazy that there is such a major disruption that prevents making new rollouts and practically paralyzing my app for over a week.

ItsBenDov avatar Apr 23 '25 18:04 ItsBenDov

Hi I'm having the same issue and no error in the logs

Hiraqui avatar Apr 24 '25 09:04 Hiraqui

I'm having the same issue, creating a new backend with demo project still not working.

BlaShadow avatar Apr 24 '25 13:04 BlaShadow

We are having the same issue. The backend was already deployed and running, we just wanted to promote a very simple change of code, build is green, rollout is not possible. No logging, no details, wasting hours to understand what is going on.

itomin avatar Apr 24 '25 16:04 itomin

The incident report claims its fixed but its not.

I am still getting failed rollouts across all projects & backends. Here is a rollout history from one of my backends proving it's the same commit and branch failing rollouts.

Image

I deleted the underlying container before re-deploying so it could re-build and rollout again. Before this recent incident I did this to fix failed rollouts and it worked perfectly fine before it started failing every time last week.

ItsTeccs avatar Apr 25 '25 01:04 ItsTeccs

Just got a reply from the support team.

They said:

We’ve identified that the changes are not being fully propagated in some regions, which is why you’re still having the issue.

I'm using asia-east and can identify the rollout is still broken.

jimmy-chiang avatar Apr 25 '25 02:04 jimmy-chiang

For some mysterious reason, it worked on a brand new backend with auto-deploy from GitHub enabled. I didn't deploy anything when creating the backend, but auto-deploy was set to true. I haven't encountered any deployment failures so far. Could it be related to manual deployments?

julian-printemps avatar Apr 25 '25 02:04 julian-printemps