coolify icon indicating copy to clipboard operation
coolify copied to clipboard

[Bug]: Error 525 SSL Handshake (Cloudflare)

Open SrJooJ opened this issue 8 months ago • 20 comments

Error Message and Logs

I've been having this error for 4 hours trying to solve it and I don't know how to solve it.

I'm literally using the same configuration that I use on all my coolify, and specifically on this domain I'm getting this error on any application I try to launch.

This specific machine (VPS) seems to be having some problem with SSL

Has anyone experienced something similar and managed to solve it?

My cloudflare is in full SSL/TLS mode and all domains are pointed to the machine.

I only have this configuration, nothing more.

Image

Steps to Reproduce

I believe it is not possible to reproduce, I do not know if it is a problem with coolify or my domain on cloudflare but it is very strange.

Example Repository URL

No response

Coolify Version

v4.0.0-beta.409

Are you using Coolify Cloud?

No (self-hosted)

Operating System and Version (self-hosted)

Ubuntu 24.04

Additional Information

I've tried regenerating the certificate, restarting the proxy, relaunching all applications, and nothing.

And the funniest thing is that if I go to the site and refresh 3 times, the site displays the interface, but if I go back after a few minutes, the error appears again.

SrJooJ avatar Apr 17 '25 23:04 SrJooJ

@SrJooJ if possible can you paste link to the site

djsisson avatar Apr 18 '25 14:04 djsisson

@SrJooJ if possible can you paste link to the site

Hi friend,

Well, yesterday I spent hours trying to solve it and with no way out, I decided to uninstall the entire Coolify, clean the instance and install everything again from scratch.

There wasn't much stuff on the machine, it was still new so it was fine. It seems to be working normally now, but I'll monitor it.

I'm uploading the applications right now, if there's an error I'll come back here. If the problem is solved I believe it was actually a coolify bug with SSL, I was taking a look at the logs before uninstalling everything yesterday and starting from scratch today and I found this:

SSL Redirect Loop

docker logs --tail 100 coolify-proxy
2025-04-18T03:02:24Z ERR Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [qdrant-agent-9.com]: error: one or more domains had a problem:\n[qdrant-agent-9.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: 2606:4700:3034::ac43:9e52: Fetching /.well-known/acme-challenge/bwlcgxLkjN4qFUdOjsRxJbe1jor8dK6vefhvn-RgWWY: Redirect loop detected\n" ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["qdrant-agent-9.com"] providerName=letsencrypt.acme routerName=https-0-fssg4wss4kokcskowkowcw48-qdrant@docker rule="Host(qdrant-agent-9.com) && PathPrefix(/)"
2025-04-18T03:02:27Z ERR Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [traefik.agent-9.com]: error: one or more domains had a problem:\n[traefik.agent-9.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: 2606:4700:3032::ac43:c1bd: Fetching /.well-known/acme-challenge/mLtWLPOmBi5wQXcx7NBQe0_pEb5rQh1e46iq6XeSDJk: Redirect loop detected\n" ACME CA=https://acme-v02.api.letsencrypt.org/directory acmeCA=https://acme-v02.api.letsencrypt.org/directory domains=["traefik.agent-9.com"] providerName=letsencrypt.acme routerName=dashboard@docker rule=Host(traefik.agent-9.com)

This is just 2 of the errors, all domains were giving this error. It was in an eternal loop and I even got a 3 hour timeout for many attempts.

SrJooJ avatar Apr 18 '25 14:04 SrJooJ

@SrJooJ please note, coolify has nothing to do with certificate generation.

if you have turned on cloudflare proxy dns, then you can no longer generate certificates using the default method.

you have to either use dns challenge or install the origin certificate yourself

djsisson avatar Apr 18 '25 14:04 djsisson

@SrJooJ please note, coolify has nothing to do with certificate generation.

if you have turned on cloudflare proxy dns, then you can no longer generate certificates using the default method.

you have to either use dns challenge or install the origin certificate yourself

I uploaded the applications and everything is working normally now.

So could this be a Cloudflare problem? It didn't make much sense, because after I cleaned it and started from scratch, it started working again.

If it was a Cloudflare issue I would still be in the same loop and it would replicate to all the machines I use Coolify on, since they also use Cloudflare without an origin certificate, I leave ports 80 and 443 open.

Could this problem return in the future if I don't install the origin certificate?

SrJooJ avatar Apr 18 '25 15:04 SrJooJ

it could also have been a clock error

djsisson avatar Apr 18 '25 15:04 djsisson

it could also have been a clock error

After a few hours the bug came back again... It's very strange, it's only on this specific machine with this domain.

I will try to install an origin certificate

SrJooJ avatar Apr 18 '25 18:04 SrJooJ

I'm back, I installed origin following the instructions: https://coolify.io/docs/knowledge-base/cloudflare/origin-cert

Even so I still have error 525, I'm running out of ideas of what to do.

SrJooJ avatar Apr 18 '25 18:04 SrJooJ

Well, I tried everything, including formatting the machine and changing from Ubuntu to Debian and still nothing.

Not even a clean install and an origin certificate saved it.

Well, I removed it from Cloudflare and I'm going to continue with this domain without being there.

If anyone has gone through the same situation and managed to solve it or if anyone has any other ideas for me to try, please leave them below.

SrJooJ avatar Apr 18 '25 21:04 SrJooJ

@SrJooJ can you provide more information on the setup of the affected servers?

That error in combination with full SSL/TLS mode would usually incidate either one of those options:

  • No valid SSL certificate installed
  • Port 443 (or another custom secure port) is not open

You already said you followed the docs to install the origin cert. Coolify is maybe still trying to serve a Let's Encrypt certificate. You might want to try deleting the acme.json from the proxy and restart it afterwards.

If you aren't using Cloudflare Tunnel, it is important that you keep port 443 open on that server. So check your firewall if you are blocking it.

Cinzya avatar Apr 21 '25 12:04 Cinzya

@SrJooJ can you provide more information on the setup of the affected servers?

That error in combination with full SSL/TLS mode would usually incidate either one of those options:

  • No valid SSL certificate installed
  • Port 443 (or another custom secure port) is not open

You already said you followed the docs to install the origin cert. Coolify is maybe still trying to serve a Let's Encrypt certificate. You might want to try deleting the acme.json from the proxy and restart it afterwards.

If you aren't using Cloudflare Tunnel, it is important that you keep port 443 open on that server. So check your firewall if you are blocking it.

Hello,

In my firewall I have the following configuration

Protocol/Port Action From
22/tcp ALLOW Anywhere
2224/tcp ALLOW Anywhere
80/tcp ALLOW Anywhere
443 ALLOW Anywhere
22/tcp (v6) ALLOW Anywhere (v6)
2224/tcp (v6) ALLOW Anywhere (v6)
80/tcp (v6) ALLOW Anywhere (v6)
443 (v6) ALLOW Anywhere (v6)

I haven't tried deleting acme.json yet, I'll try it.

But as I mentioned above, I tried to do a complete clean installation, I even changed the operating system.

When I leave the domain out of Cloudflare it works perfectly, when I bring it to Cloudflare it starts giving errors in all applications.

The funny thing is that with just this specific domain, I have about 10+ domains in Cloudflare that don't have this problem and have the same coolify configuration.

I don't know if it's the VPS machine or CloudFlare.

In Cloudflare, SSL/TLS is in full mode and the domain is pointing to the machine correctly.

SrJooJ avatar Apr 23 '25 18:04 SrJooJ

did this ever get solved? im getting the same issue!

edit: turns out letsencrypt is on planned maintenance rn lol

edit edit: letsencrypt is back up but new subdomains still return 525 welp

lenzfliker avatar Jun 05 '25 19:06 lenzfliker

did this ever get solved? im getting the same issue!

edit: turns out letsencrypt is on planned maintenance rn lol

edit edit: letsencrypt is back up but new subdomains still return 525 welp

Sadly not

I tried everything.

In the end I ended up abandoning it and this machine I have here is the only one that won't use coolify

There seems to be a problem with the specific domain I allocated. All the others are working perfectly.

SrJooJ avatar Jun 05 '25 22:06 SrJooJ

did this ever get solved? im getting the same issue! edit: turns out letsencrypt is on planned maintenance rn lol edit edit: letsencrypt is back up but new subdomains still return 525 welp

Sadly not

I tried everything.

In the end I ended up abandoning it and this machine I have here is the only one that won't use coolify

There seems to be a problem with the specific domain I allocated. All the others are working perfectly.

So I did find a workaround of sorts.. not ideal but seems to be working!

Before you click 'deploy' on the app/service in coolify, copy the domain and add A record on cloudflare as a new subdomain with proxy status turned OFF (wildcard doesnt count add the specific subdomain)

Deploy the app/service and visit the site once. You can now turn ON the proxy in cloudflare to check if the site is still resolving - it should at least it did for me. Feel free to delete that A record.

Not sure how this thing will affect app/service to DB connections but im not looking forward to it lol

PS. I'm on caddy proxy so i dont need the origin cert - crashed my coolify trying to install rookie mistake lol saved thanks to ssh. I didnt delete acme.json. Ports are open as needed.

lenzfliker avatar Jun 06 '25 10:06 lenzfliker

@lenzfliker the lets encrypt certificate lasts 90 days, so you will need to redo this every couple of months, since you can't use http challenge from behind a dns proxy.

you can switch to dns challenge adding labels to the caddy proxy:

labels:
  - caddy.acme_dns: "cloudflare {env.CF_API_TOKEN}"
  - caddy.email: "{env.EMAIL}"

or i assume you can do the same adding the certificate in the labels

maybe something like

labels:
  -  caddy.tls: /path/cert.pem /path/cert.key

but unsure on this last one, someone who knows caddy can prob tell you the correct label to use here

djsisson avatar Jun 06 '25 11:06 djsisson

@lenzfliker the lets encrypt certificate lasts 90 days, so you will need to redo this every couple of months, since you can't use http challenge from behind a dns proxy.

you can switch to dns challenge adding labels to the caddy proxy:

labels:

  • caddy.acme_dns: "cloudflare {env.CF_API_TOKEN}"
  • caddy.email: "{env.EMAIL}"

or i assume you can do the same adding the certificate in the labels

maybe something like

labels:

  • caddy.tls: /path/cert.pem /path/cert.key

but unsure on this last one, someone who knows caddy can prob tell you the correct label to use here

Been trying to set this up for like 2h now can't seem to pull the caddy with cloudflare to proceed im on 2.8.4

lenzfliker avatar Jun 06 '25 20:06 lenzfliker

@lenzfliker

you have to build your own including both plugins, but its simple to do, takes a couple of mins

run this:

cat <<EOF | docker build -t custom-caddy:2.10 -
FROM caddy:2.10-builder AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/cloudflare \
    --with github.com/lucaslorentz/caddy-docker-proxy/v2

FROM caddy:2.10

COPY --from=builder /usr/bin/caddy /usr/bin/caddy
EOF

then once its built , change the compose in coolif to point to the new image you made, in this case it would jsut be custom-caddy:210

change build and tag as required

djsisson avatar Jun 06 '25 21:06 djsisson

@lenzfliker

you have to build your own including both plugins, but its simple to do, takes a couple of mins

run this:

cat <<EOF | docker build -t custom-caddy:2.10 -
FROM caddy:2.10-builder AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/cloudflare \
    --with github.com/lucaslorentz/caddy-docker-proxy/v2

FROM caddy:2.10

COPY --from=builder /usr/bin/caddy /usr/bin/caddy
EOF

then once its built , change the compose in coolif to point to the new image you made, in this case it would jsut be custom-caddy:210

change build and tag as required

I tried again and it fails to build

2025/06/07 08:02:42 [INFO] exec (timeout=0s): /usr/local/go/bin/go build -o /usr/bin/caddy \
    -ldflags "-w -s -trimpath" -tags nobadger,nomysql,nopgx

# github.com/caddyserver/caddy/v2
/go/pkg/mod/github.com/caddyserver/caddy/[email protected]/context.go:541:12: undefined: zapslog.HandlerOptions

2025/06/07 08:03:54 [INFO] Skipping cleanup as requested; leaving folder intact: /tmp/buildenv_2025-06-07-0802.3653704630
2025/06/07 08:03:54 [FATAL] exit status 1

------
failed to solve: process "/bin/sh -c xcaddy build \
    --with github.com/caddy-dns/cloudflare \
    --with github.com/lucaslorentz/caddy-docker-proxy/v2" did not complete successfully: exit code: 1

tried grabbing zap as well but same issue

is it because im on caddy 2.8.4? do i need 2.10?

lenzfliker avatar Jun 07 '25 08:06 lenzfliker

can you show me what command you ran?

you would need to pick an older cloudflare dns, since the newest one only works on 2.10, due to libdns changes in caddy

djsisson avatar Jun 07 '25 08:06 djsisson

can you show me what command you ran?

you would need to pick an older cloudflare dns, since the newest one only works on 2.10, due to libdns changes in caddy

  1. Added a new Dockerfile inside root@ubuntu-8gb-fsn1-1:/data/coolify/proxy/caddy with your code and my caddy version
  2. Updated the docker-compose.yml to use this custom image
  3. docker compose down
  4. docker compose build caddy
  5. docker compose up -d
  6. ! caddy Warning pull access denied for custom-caddy
  7. continues to download and install then throws above pasted error

lenzfliker avatar Jun 07 '25 16:06 lenzfliker

I would just run the build command as is in the terminal

But if you want to run it from a compose what image name did you use, as it had to be your own local tag

And it won't run if it doesn't build, and it won't build if you choose anything other than 2.10 unless you pick an older cloudflare dns digest

djsisson avatar Jun 07 '25 18:06 djsisson

@SrJooJ I had the same issue here and solve it. Just was a typo instead of https://mydomain.com I put https:/my domain.com forgot the one "/" Now everything works great

Dokja620 avatar Aug 15 '25 12:08 Dokja620

I'm going to assume that this is fixed, as one user shared a solution and others have stopped responding. If this is still a problem for other people, we can re-open it again.

Cinzya avatar Nov 02 '25 00:11 Cinzya