coollabsio
[Bug]: Double Github dedployments on push
Description
I have been experiencing an issue, where I get double Deployment triggers from GitHub webhooks so my applications are restarting twice. Any ideas as to what is going on and how to fix it? It has been going on over a month, I am just not getting to posting about this. Image examples below.
Additionally, due to this the application in many cases ends up in an exited or Stopped state I am assuming due to the rapid attempts to redeploy simultaneously.
My Env: total 4 VPS' 1 is the master and 3 are docker engine servers created using the v4 platform. Across all of the servers and projects there are probably 30-45 resources. I am what would be considered a power user, which is why i love sponsoring the project and which is also why I encourage many of my clients to use coolify and sponsor as well.
https://forthe.meme/ss/msedge_DPNzzZ6eLt.png https://forthe.meme/ss/msedge_EHl3Uchc2Q.png
Minimal Reproduction (if possible, example repository)
https://forthe.meme/ss/msedge_DPNzzZ6eLt.png https://forthe.meme/ss/msedge_EHl3Uchc2Q.png
I have also been informed in my forum post in the discord that others have the same issue.
I should note that my config is the recommended host vps running on the panel and then docker engine servers running the resources which run fine except for this issue.
Exception or Error
No response
Version
4.0.0-beta.309
If you open the Source tab, do you have only one GitHub App source?
If yes, please open that GitHub App (click on Update repositories), on GH, click on App Settings -> Advanced and check if there are double webhooks sent.
My only guess for this problem (as I cannot reproduce), is that you have at least 2 GH Apps (or one github app and one manual webhook for your repository) configured and both sending a webhook.
Thank you for your response.
I reviewed and in 1 of the Teams I do indeed have 2 sources however, the deliveries are not in both sources. I do however have 2 separate teams that have the same github source connected and I am seeing the same recent deliveries in. The reason for 2 teams connecting to the same github is simply because i need to be able to separate the permissions and teams is the only way to do that currently.
Please let me know if this makes sense...
2 Teams both have a source connected to this github.com account. Also for reference, I cannot make this account a global source as I do have a third team that this github acct isn't linked to.
This does seem to have fixed it. I am not sure if this is possible. However, If you could add a warning pop up or disclaimer when someone tries to link a source that is already linked, If it can then give them a warning that they should go to the sources webhook settings and disable it for the duplicate source. As there are many cases where 1 might connect the same source twice across multiple teams.
Same issue here, triggers double so sometimes it would cause the server to die (since it is a low spec) doing builds.
The issue is coming from having two or more teams (needed for permissions), each team have its own GitHub app in sources.
- the 'root' team had access to all repositories in the org.
- the frontend team had access to the frontend repository (the app) only.
What I have observed is the root team doesn't have the app deployed, it is only deployed in the front team. But being the same url for webhook from GitHub, it triggers twice, one for every app that has access to that repository.
The quickest solution I found is to narrow the permissions of the 'root' team to exclude the frontend repository. And now it works perfectly, one single build and deploy.
I am not very familiar with coolify's code or inner workings, but it must be handling something inside to do deploys for every single github (github app in sources) that has access to the repository being pushed.
I am willing to help a little in testing possible fixes at least.
Same issue here, triggers double so sometimes it would cause the server to die (since it is a low spec) doing builds.
The issue is coming from having two or more teams (needed for permissions), each team have its own GitHub app in sources.
- the 'root' team had access to all repositories in the org.
- the frontend team had access to the frontend repository (the app) only.
What I have observed is the root team doesn't have the app deployed, it is only deployed in the front team. But being the same url for webhook from GitHub, it triggers twice, one for every app that has access to that repository.
The quickest solution I found is to narrow the permissions of the 'root' team to exclude the frontend repository. And now it works perfectly, one single build and deploy.
I am not very familiar with coolify's code or inner workings, but it must be handling something inside to do deploys for every single github (github app in sources) that has access to the repository being pushed.
I am willing to help a little in testing possible fixes at least.
The only and best fix is to goto the other apps on the instance on GitHub and disable the webhook calls. This will fix the issue.
I ran into a very similar issue, because I needed to have a shared server between two businesses, so I added ourselves onto a shared team with access to that server. Disabling one of the two webhooks associated with the two apps in Github created (one for each team) fixed the issue for me.
(SIDE NOTE: It would be amazing to enable the sharing of a server to another team)
There is ability to make sources global but specific would be better. I’d also like project level access permissions. Sent from my iPhoneOn Aug 31, 2024, at 2:52 PM, Nick Maietta @.***> wrote: I ran into a very similar issue, because I needed to have a shared server between two businesses, so I added ourselves onto a shared team with access to that server. Disabling one of the two webhooks associated with the two apps in Github created (one for each team) fixed the issue for me. (SIDE NOTE: It would be amazing to enable the sharing of a server to another team)
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>
I had the same problem with double deployments freezing my VPS. My fix: Only using one github app for auto deployment.
Happening here as well (dockerfile deployment).
I do have two different Github Apps, based on teams (since the System Wide checkbox isn't working #2643), as well as having the project being deployed on two environments (with same build arguments, so a different build isn't actually required), resulting in 4 deployments at each push.
Ran into the same issue. I am unable to use a single github source system wide across teams. Checking the "system wide" option does not work, and so i am forced to use two github apps, and this is causing dual deployments on push.
You need to go in to github and disable the webhooks on one of the 2 sources registered on github. It is because github is sending 2 webhooks and coolify isn't registering that it has already received a request to redeploy on that github head
I'd say this is rather a bug / missing feature than a misconfiguration. I have one instance with two teams. Team A has a GitHub app with access to all repositories and an active deployment of my-favourite-app (development branch) Team B has another GitHub app - for the same GitHub acccount - but with limited access to selected repositories. It also has an active deployment of my-favourite-app (main branch).
Since there's an overlap of repositories accessible to both teams, it seems that both apps trigger the webhook and then the instance handles both webhooks for both teams at the same time.
I would need to either disable one of the deployments or not use the teams feature and setup a second Coolify instance for team b.
So I'd be more than happy if a single instance could properly manage webhooks for multiple teams.
New observation from today (v4.0.0-beta.390 ; context: Coolify teams A / B with access to repos from the same Github account via two separate GitHub apps - as in my comment just above):
- Team B has a new GIt-app-based deployment of
my-new-favourite-app - Team A doesn't
git push triggers two simultaneous deployments of my-new-favourite-app for Team B.
Does it help you to know that although I don't see double deployments in my instance, I receive TWO emails confirming every successful deployment?
EDIT: Nevermind. It was a configuration error on my side. Sorry.
I will add a few fixes in v408 for this. It will prevent double deployments for the same commit through UI/API/Webhook.