cloudflared icon indicating copy to clipboard operation
cloudflared copied to clipboard

🐛Access tunnel via load balancer fails.

Open jonface opened this issue 2 years ago • 0 comments

DNS Setup.

Type / Name / Content

CNAME, server-1.domain.com , xxxxx.cfargotunnel.com CNAME, server-2.domain.com , yyyyy.cfargotunnel.com

Load Balancer Config

Load balancer running on loadbalancer.domain.com Load balancer pool containing server-1 and server-2 . All health checks are passing.

Configure the pool origins to set/not set the HOST header, it makes no difference at all, as far as I can tell they are only passed to the underlying application (eg the web server, not used by cloudflared or the load balancer).

Now set the tunnel catchall for server-1 to reply with a 501 error, server-2 to a 502 error, this helps to work out which server is getting the request.

Set cloudflared LOGLEVEL to VERBOSE.

The tunnel local application routing can be anything as long as there is a suitable HTTP connection to avoid errors in the cloudflared logs, eg,

Server-1 , http://localhost:80, Server-2 http://localhost:80 , run a basic web server, netcat, anything to allow this to pass.

At this point we have a load balancer setup with a pool of 2 origins, both origins are healthy, can be accessed directly via their tunnel dns cname server-1.domain.com and server-2.domain.com .

Now try and access via the load balancer. HTTP status will either be 501 or 502.

Check the cloudflared logs, you can see the request coming in, but the domain is set to loadbalancer.domain.com , there is no rule to match that and so cloudflared returns the catchall 50x error.

Setting the host header from the origin pool config has no effect.

I experimented adding the auto generated cfargotunnel DNS values into the pool instead of server-1 and server-2 and it started to work. BUT, after a couple of requests, I started getting intermittant 50x errors. I checked the logs and the request for server-1 was going to server-2 or vice versa, sometimes. At this point I gave up, way outside of my control.

I can’t be the only one who thought I would secure my servers from the outside world via tunnels and try and put them in a load balancer?

Thanks for any advice.

Log below shows a request coming in via the load balancer to one of the servers but it does not match any of the rules and so the catchall is triggered.

Mar 14 09:01:43 server-1 cloudflared[630]: 2023-03-14T09:01:43Z DBG CF-RAY: 7a7b556cb85db2eb-MAN GET https://loadbalancer.domain.com/ HTTP/1.1 Mar 14 09:01:43 server-1 cloudflared[630]: 2023-03-14T09:01:43Z DBG Inbound request CF-RAY=7a7b556cb85db2eb-MAN Header="map[Accept:[ Cf-Ray:[7a7b556cb85db2eb-MAN]" connIndex=3 host=loadbalancer.domaincom path=/ rule=3 Mar 14 09:01:43 server-1 cloudflared[630]: 2023-03-14T09:01:43Z DBG CF-RAY: 7a7b556cb85db2eb-MAN Request content length 0 Mar 14 09:01:43 server-1 cloudflared[630]: 2023-03-14T09:01:43Z DBG CF-RAY: 7a7b556cb85db2eb-MAN Status: 501 Not Implemented served by ingress 3 connIndex=3 Mar 14 09:01:43 server-1 cloudflared[630]: 2023-03-14T09:01:43Z DBG CF-RAY: 7a7b556cb85db2eb-MAN Response Headers map[] connIndex=3 Mar 14 09:01:43 server-1 cloudflared[630]: 2023-03-14T09:01:43Z DBG CF-RAY: 7a7b556cb85db2eb-MAN Response content length 0 connIndex=3

jonface avatar Mar 14 '23 09:03 jonface