[Bug]: Terminal not accessible

[dickermoshe]

Error Message and Logs

I've installed coolify on a spare laptop. I used Cloudflare tunnels to access it remotely. I set the correct environmental vars so that the the realtime and terminal api should work on the frontend.

REALTIME websockets work, But TERMINAL websockets do NOT!

Steps to Reproduce

1. Install Coolify on a Server
2. Navigate to Terminal Page ERROR: Terminal websocket connection lost.

Ok, let's use curl on localhost to see what's going on:

root@oldlaptop:/home/mick# curl localhost:6002/ready
OK

Seems like the server for the realtime API is running, that's good

Let's get those logs:

root@oldlaptop:/hodocker logs 4bb1497b23c9 -tv
2024-12-15T02:28:25.311313459Z 2024-12-15 02:28:25 [TERMINAL] Coolify realtime terminal server listening on port 6002. Let the hacking begin!
2024-12-15T02:28:25.576541725Z 2024-12-15 02:28:25 [SOKETI]
2024-12-15T02:28:25.584539981Z 2024-12-15 02:28:25 [SOKETI] 🕵️‍♂️ Initiating metrics endpoints...
2024-12-15T02:28:25.585333414Z 2024-12-15 02:28:25 [SOKETI]
2024-12-15T02:28:25.586187631Z 2024-12-15 02:28:25 [SOKETI] 🎉 Server is up and running!
2024-12-15T02:28:25.587015263Z 2024-12-15 02:28:25 [SOKETI] 📡 The Websockets server is available at 127.0.0.1:6001
2024-12-15T02:28:25.587567900Z 2024-12-15 02:28:25 [SOKETI] 🔗 The HTTP API server is available at http://127.0.0.1:6001
2024-12-15T02:28:25.588409286Z 2024-12-15 02:28:25 [SOKETI] 🎊 The /usage endpoint is available on port 9601.
2024-12-15T02:28:25.588987941Z 2024-12-15 02:28:25 [SOKETI]
2024-12-15T02:28:25.589690367Z 2024-12-15 02:28:25 [SOKETI] (node:8) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-12-15T02:28:25.590229087Z 2024-12-15 02:28:25 [SOKETI]
2024-12-15T02:28:25.590810387Z 2024-12-15 02:28:25 [SOKETI] Please migrate your code to use AWS SDK for JavaScript (v3).
2024-12-15T02:28:25.591389923Z 2024-12-15 02:28:25 [SOKETI] For more information, check the migration guide at https://a.co/7PzMCcy
2024-12-15T02:28:25.592023949Z 2024-12-15 02:28:25 [SOKETI] (Use `node --trace-warnings ...` to show where the warning was created)

Hmmm, looks like everything is ok. Let's try copying the request from chrome and seeing if it works from the CLI

We can use a tool like websocat to test if this works locally:

root@oldlaptop:/home/mick# websocat 'ws://localhost:6002'   -H 'Upgrade: websocket'   -H 'Origin: https://coolify.dickersystems.com'   -H 'Cache-Control: no-cache'   -H 'Accept-Language: en-US,en;q=0.9'   -H 'Pragma: no-cache'   -H 'Connection: Upgrade'   -H 'Sec-WebSocket-Key: RB72t/uvO8ejOMLhMQljUA=='   -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'   -H 'Sec-WebSocket-Version: 13'   -H 'Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits'
websocat: WebSocketError: WebSocketError: Received unexpected status code (400 Bad Request)
websocat: error running

Hmmm, it doesn't. No logs in docker logs either. Seems like we need better logging!!

We can confirm that copything from curl SHOULD work by copying the request from the realtime websocket that is working.

root@oldlaptop:/home/mick# websocat 'ws://localhost:6001/app/REDACTED?protocol=7&client=js&version=8.3.0&flash=false'   -H 'Upgrade: websocket'   -H 'Origin: https://coolify.dickersystems.com'   -H 'Cache-Control: no-cache'   -H 'Accept-Language: en-US,en;q=0.9'   -H 'Pragma: no-cache'   -H 'Connection: Upgrade'   -H 'Sec-WebSocket-Key: +has5XTnkQrT1iRzO6Lx3g=='   -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'   -H 'Sec-WebSocket-Version: 13'   -H 'Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits'
{"event":"pusher:connection_established","data":"{\"socket_id\":\"REDACTED\",\"activity_timeout\":30}"}

Example Repository URL

No response

Coolify Version

v4.0.0-beta.379

Are you using Coolify Cloud?

No (self-hosted)

Operating System and Version (self-hosted)

Ubuntu 24.04.1 LTS

Additional Information

No response

[ricky-cc]

+1 got the same issue with custom ssh port

[gitsacha]

+1, I've been able to fix the ws connection error for the realtime service using this thread but I wasn't able to find a solution for the terminal ws.

Using Cloudflare Tunnels (cloudflared) with the following config on an ec2:

credentials-file: /root/.cloudflared/_TUNNEL_ID_.json
ingress:
  - hostname: "ssh.mydomain.com"
    service: ssh://localhost:22
  - hostname: "realtime.mydomain.com"
    service: http://localhost:6001
  - hostname: "coolify.mydomain.com"
    service: http://localhost:8000
  - hostname: "*.mydomain.com"
    service: http://localhost:80
  - service: http_status:404

[KwaminaWhyte]

Having same issue

[nvo87]

Hi! I've solved these issue by setting path to /terminal/ws in cloudflare tunnel.

We can see that browser tries to request wss://yourdomain.com/termial/ws

So we need to connect it with 6002 port of coolify-realtime container

P.S. I think the instruction terminal.coolify.io mapped to localhost:6002 from documentation is misleading. To make requests to terminal.coolify.io you need to add TERMINAL_HOST=terminal.coolify.io to /data/coolify/source/.env. But this will take longer and won't work anyway, because /terminal/ws endpoint is bound to app.coolify.io.

[rber474]

Hi!

Had the same issue and applied @nvo87 solution. Now it is working. Please, note host order is important, the host <yourdomain.ext>/terminal/ws should be ordered before the host pointing to port 8000.

[gitsacha]

Thanks! It worked for me too.

For those using cloudflared with a config file directly on the server (typically at /etc/cloudflared/config.yml or ~/.cloudflared/config.yml), here's the fix:

tunnel: YOUR_TUNNEL_ID
credentials-file: /root/.cloudflared/YOUR_TUNNEL_ID.json
ingress:
  - hostname: "ssh.yourdomain.com"
    service: ssh://localhost:22
  - hostname: "realtime.yourdomain.com"
    service: http://localhost:6001
  # FIX HERE
  - hostname: "coolify.yourdomain.com"
    service: http://localhost:6002
    path: "/terminal/ws"
  - hostname: "coolify.yourdomain.com"
    service: http://localhost:8000
  - hostname: "*.yourdomain.com"
    service: http://localhost:80
  - service: http_status:404

[cesarhfborges]

Hello, I'm having the same problem, the socket is not working, neither using a local ip nor using a domain, it doesn't connect to port 6001 or 6002, I can't say if it happened after I updated to version v4.0.0-beta.390 but it was working both via local ip and domain but I needed to connect to a certain container and that's when I noticed.

I've already tried to point cloudflared directly to the address + port or just to the address without a port, anyway, but it seems to me something happened with the coolify socket service and I believe that it itself is not responding.

[HuakunShen]

Hi!

Had the same issue and applied @nvo87 solution. Now it is working. Please, note host order is important, the host <yourdomain.ext>/terminal/ws should be ordered before the host pointing to port 8000.

This is so helpful, I couldn't get /terminal/ws working following the documentation at https://coolify.io/docs/knowledge-base/cloudflare/tunnels/single-resource#tunnel-coolify Changing the order solves the problem for me

Using this order from the documentation doesn't work

@andrasbacsai Can you update the docs and mention the /terminal/ws one should come before the * path one?

Another thing I noticed was nested subdomains doesn't seem to work for realtime. e.g. realtime.coolify.domain.com doesn't work, but realtime-coolify.domain.com works.

[Cinzya]

@HuakunShen thanks for pointing it out to us. We have a dedicated repository for the docs, if you can open an issue on there directly next time, then we are able to fix this much quicker. We have a dedicated team working on the docs that don't read everything in here and Andras is usually very busy.

@KwaminaWhyte @dickermoshe @cesarhfborges If you guys can take another look at the Coolify Docs and double check your Cloudflare settings that there is nothing missing on your end.

If there are no more comments about this otherwise, I will then close this issue soon. As it seems it was some misconfiguration on the Users end, partially to misleading Docs.

[ErlandssonA]

I am having the same problem with the terminal. But I am running coolify on a VM on my dedicated server that's running proxmox. I only have 1 public IP so I use nginx as a reverse proxy on the dedicated server for all applications etc.

I think the problem is laying in my nginx configuration for coolify. But I am not sure on how to fix it.

[andrasbacsai]

I have made a few improvement in #5815 that will help in a few cases with CF Tunnels.

Basically what I've tested it works (lol).

[Cinzya]

@ErlandssonA there is already a thread in https://github.com/coollabsio/coolify/discussions/3097#discussioncomment-12793669 on how to configure Nginx proxy manager correctly for the Terminal and WebSocket to work properly.

[ErlandssonA]

@Cinzya Ohh I haven't seen that! Thank you.

I will take a look at it!

[ErlandssonA]

@ErlandssonA there is already a thread in #3097 (reply in thread) on how to configure Nginx proxy manager correctly for the Terminal and WebSocket to work properly.

I tried this but I still couldn't get it to work. Would it be any different if im running coolify on a VM and using nginx as a reverse proxy for my ProxMox server (a dedicated server with 1 public IP)

[magnusburton]

This has stopped working again. I have my Cloudflare config as described and I’ve gotten it to work previously (don’t remember the exact version of the regression). I’m on beta 428.