coolify icon indicating copy to clipboard operation
coolify copied to clipboard

[Bug]: 504 Gateway Time-out when updating application general configuration

Open carbontwelve opened this issue 1 year ago • 26 comments

Description

I have a public git repository that is using the Dockerfile buildpack. When editing any of the general configuration values, upon pressing save, after a while a full page modal pops up with 504 Gateway Time-out error.

Minimal Reproduction (if possible, example repository)

  • Create a new project from public git repository: https://github.com/photogabble/website/
  • Update to use the release/2024 branch (as this has the new Dockerfile)
  • Deploy
  • Realise that its defaulted to port 3000 on creation and update it to port 80, click save and wait for the 504 Gateway Time-out

I then reload the page and see the The latest configuration has not been applied toast, so the configuration change was made.

Exception or Error

No response

Version

v4.0.0-beta.315

carbontwelve avatar Jul 17 '24 23:07 carbontwelve

I cannot reproduce the issue. Are you using the cloud version?

andrasbacsai avatar Jul 18 '24 10:07 andrasbacsai

I'm on self hosted, on a 16-core dedicated server.

Have since restarted the coolify containers and still getting the same thing happen: image

It's a 504 in response to a post request made to /livewire/update after 60 seconds. I have attached a shell to the coolify container and see nothing pop into the laravel.log so I believe it's taking longer to process the update than the 60 second timeout set in nginx.

As an aside, it might be a good idea to add handling for this error:

Livewire.onError((message, statusCode) => {
    if (statusCode == 504) {
        // Handle the timeout here, then return false
        return false;
    }
});

carbontwelve avatar Jul 18 '24 11:07 carbontwelve

I've got this issue too on selfhosted version.

FrSenpai avatar Aug 05 '24 22:08 FrSenpai

same here when trying to stop the appwrite stack

fr0ntd3v avatar Aug 15 '24 16:08 fr0ntd3v

Same here. When I'm trying to build my next js app

calebrussel77 avatar Aug 17 '24 21:08 calebrussel77

Add a swapfile resolved my issue.

https://support.hostinger.com/en/articles/8124185-how-to-set-up-swap-on-hostinger-vps

FrSenpai avatar Aug 18 '24 12:08 FrSenpai

I believe #3220 is related to this issue.

I am also struggling with the exact same issue and trying to diagnose this. Specifically in the default localhost instance

image

james090500 avatar Aug 28 '24 16:08 james090500

having the same error on beefy server when I save configuration

orenaksakal avatar Sep 29 '24 01:09 orenaksakal

Same here while using Hetzner server and trying to build a SvelteKit app.

nopale avatar Sep 30 '24 14:09 nopale

Same here. image

thansil-emst avatar Oct 01 '24 08:10 thansil-emst

== Self hosted ==

Same issue when trying to connect with remote server. Getting this issue at "Start proxy" step.

Any solution ?

=== Update ===

image

In my case i got Gateway timeout issue during the onboarding process (Proxy setup stage).

I have chosen "remote server". Server SSH connection established but the proxy start was not working. After a long try i found that coolify is trying to pull and start the traefik docker container image into my remote server and the container runs on port 80.

unfortunately I had nginx sever which is listening on port 80 and due to that traefik container failed to start. In that case the timeout exception occurs and got 504 error.

I have stopped the nginx server and tried again. Now the proxy connection established and able to do the deployment.

Make sure the port 80 is open in the remote server or change the traefik default port from 80 to different port.

I hope this helps.

ATS-CEO avatar Oct 01 '24 19:10 ATS-CEO

I added swap memory to my digital ocean vps and worked.

https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-20-04

lesimoes avatar Oct 05 '24 22:10 lesimoes

same here. Anyone solution?

1oid avatar Nov 07 '24 04:11 1oid

I added swap memory to my digital ocean vps and worked.

https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-20-04

@1oid ☝️

Follow this

thansil-emst avatar Nov 07 '24 11:11 thansil-emst

I am experiencing the same behavior with my Coolify instance on my dedicated server. I just checked the coolify logs.

2024/11/09 12:18:00 [error] 158#158: *1069966 upstream timed out (110: Unknown error) while reading response header from upstream, client: 172.18.0.6, server: _, request: "POST /livewire/update HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "REDACTED", referrer: "https://REDACTED/project/fo0cso8cogok8o4g888s8o0c/production/application/t8kkoo0kwscko8c0s04sg0gg?deployment_uuid=ecoko0k4gsow4cwws0kcksog"

webertim avatar Nov 09 '24 12:11 webertim

I added swap memory to my digital ocean vps and worked. https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-20-04

@1oid ☝️

Follow this

This works, I don't know why though

herdeybayor avatar Nov 21 '24 20:11 herdeybayor

I guess if the OS is out of physical memory, linux might employ the OOM Killer.

eyyyyyyy3 avatar Nov 21 '24 20:11 eyyyyyyy3

for me adding swap didn't help but disabling hetzner firewall works I already allow all realtime ports in that firewall but when enabled it 504s. I keep ufw and disable hetzner firewall to overcome this problem

orenaksakal avatar Dec 01 '24 00:12 orenaksakal

I added swap memory to my digital ocean vps and worked.

https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-20-04

Yeah this worked for me also.

JosefSaltz avatar Feb 14 '25 07:02 JosefSaltz

Nothing suggested above solves it for me, takes around 2 minutes to change a simple domain in the resource's general settings.

Fresh install, even tried PHP_MEMORY_LIMIT=1024M, no luck.

Server is beefy with no usage at all, 1% cpu usage - 8c/16T, mem 51g available/64g, plenty of ssd space.

hemant6488 avatar Feb 15 '25 10:02 hemant6488

I'm also seeing this an a very high spec freshly installed server..

richardARPANET avatar Mar 06 '25 07:03 richardARPANET

I was stuck on this when I activated automatic deploy for github PR.

Something that worked for me:

free -h  # check your swap, for me it was Swap: 0B 0B 0B

sudo fallocate -l 2G /swapfile  # Create a 2 Go folder
sudo chmod 600 /swapfile  # make it safe
sudo mkswap /swapfile  # Init the swap
sudo swapon /swapfile  # Activate the swap
echo "/swapfile none swap sw 0 0" | sudo tee -a /etc/fstab  # persist it

free -h  # check your swap again, for me it was Swap: 2.0Gi 0B 2.0Gi

balibou avatar Mar 11 '25 09:03 balibou

You can use this one liner to create SWAP memory.

curl -sSL https://gist.githubusercontent.com/mrcodekiddie/23d69bbc3cd09875e23704d5cdc40f64/raw/cda2c61e24b96550e700a4e3293aa798284e72eb/swap-creator.sh | bash

dbjpanda avatar Apr 01 '25 20:04 dbjpanda

thank you guys. since i m running on proxmox server, here is what i needed to do inside of my coolify VM:

  1. Access the VM:

    • Open the Proxmox web interface.
    • Open a console window for the VM or SSH into the VM directly.
  2. Create a Swap File:

    • Use the following command to create a swap file. Replace 2G with the desired swap size.
      sudo fallocate -l 2G /swapfile
      
  3. Secure the Swap File:

    • Change the permissions to make the swap file readable only by root.
      sudo chmod 600 /swapfile
      
  4. Mark the File as Swap Space:

    • Set up the swap area with the following command:
      sudo mkswap /swapfile
      
  5. Enable the Swap File:

    • Activate the swap file:
      sudo swapon /swapfile
      
  6. Persist the Swap File:

    • Add the swap file to /etc/fstab to make it permanent. Use a text editor to add the following line:
      /swapfile none swap sw 0 0
      
  7. Verify the Swap Space:

    • Check that the swap is active with:
      sudo swapon --show
      

leikoilja avatar Apr 06 '25 11:04 leikoilja

I also created a 2gb swap memory. Around 2am the website went down with the Gateway Timeout. I deactivated Sentinel because it was out of sync. After that the Gateway Timeout was suddenly gone.

Lets see if that fixed it for me, time will tell.

Edit: The website went down May 6, 2025, 02:06:13. Three weeks without a problem.. Thats weird, ill try to find something in the logs later

lorenzrabs avatar Apr 17 '25 07:04 lorenzrabs

@orenaksakal I have exactly the same problem with Hetzner Firewall. I solved it by enabling all TCP and UDP outgoing traffic. Check you have UDP enabled to the outside too!

boriel avatar Jun 01 '25 13:06 boriel

I added swap memory to my digital ocean vps and worked.

https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-20-04

up

pilgerj avatar Jul 21 '25 19:07 pilgerj

Thanks to adding a swap file, I was able to fix the problem. I set it to 4 GB because the swap usage was peaking around 2.7 GB. The application in question is a fairly heavy Next.js project, which explains the high memory demand.

AlbertoCruzLuis avatar Oct 19 '25 10:10 AlbertoCruzLuis