Default Traefik not working properly on Rancher Desktop because not routable IP to the instance

[dragoicaru]

• Right now all ports above 1023 are auto-forwarded to localhost, but for privileged ports this has to be done manually. Right now the default Traefik installed with Rancher Desktop does not really work properly due to this; will be good to have a routable IP address for the instance so we can make Traefik work properly and use the corresponding ports for HTTP/HTTPS (80, 443).

[mattfarina]

Thanks for bringing this up. We are aware of the problem and already working on a solution.

[thehejik]

The forwarding of the Traefic Ingress to ports 80 and 443 on localhost works fine for me on MacOS Catalina but doesn't work on Linux openSUSE 15.2. Found when testing Rancher Desktop 0.6.0.

On Linux user has to find the Ingress port manually first on localhost by for. eg:

kubectl get svc -n kube-system traefik -o json | jq '.spec.ports[1].nodePort'

(where .spec.ports[1] stands normally for HTTPS)

Then he might access the Ingress in a Browser by using URL https://localhost:given_port

[jandubois]

On Linux user has to find the Ingress port manually first on localhost by for. eg:

The root cause is the same between macOS and Linux, but the fixes will have to be different.

First localhost: this works on macOS because we can bind privileged ports to 0.0.0.0 as a regular user, whereas on Linux you would have to be root (or at least have the NET_BIND capability).

On macOS we should have a routable port once the vde_vmnet work is completed.

On Linux we will need to create a tap device for this, and we don't have that work scheduled yet. So this will not happen in time for the next release.

[gaktive]

@adamkpickering can you check if a 0.0.0.0 workaround for localhost on Linux works? We can document a port forward section in Lima (default is: 127.0.0.1), then try accessing port 443. We can then update the release notes, though @agracey may have another workaround to consider (att'n: @btat)

[adamkpickering]

Ok, to clear up some confusion:

• The default IP address that the port forwards are bound to is 0.0.0.0, and has been since at least version 0.6.0.
• On Linux you cannot bind to ports below 1024 unless, as Jan says, you are root or have the NET_BIND capability. This is true whether we're talking about binding on 0.0.0.0 or 127.0.0.1.
• Our current experience is similar to minikube, where you have to run a command like minikube service <name> --url in order to learn what port to access your service on. It would be nice to be able to just use ports 80 and 443 on Linux, but this needs some work. I don't see why we can't do it, but it's definitely a task for the future.

[jandubois]

#964 tracks creating a routable IP address on Linux

[thehejik]

This can be used as workaround on Linux with installed RD for accessing Traefik Ingress on https://localhost/ in browser:

$ nodeport=$(kubectl get svc -n kube-system traefik -o jsonpath='{.spec.ports[1].nodePort}')
$ sudo iptables -t nat -A OUTPUT -o lo -d 127.0.0.1 -p tcp --dport 443 -j DNAT  --to-destination 127.0.0.1:$nodeport

In case you need to access Ingress also from different machine by using URL like https://my-linux-rd-machine/ you may add:

$ echo 1 | sudo tee /proc/sys/net/ipv4/conf/all/route_localnet
$ sudo iptables -t nat -A PREROUTING -p tcp --dport 443 -j DNAT --to-destination 127.0.0.1:$nodeport

Tried on openSUSE 15.3 with disabled firewalld service... (iptables using ACCEPT policy)

[jetersen]

Nowadays anything .localhost should route to 127.0.0.1 so myingress.rancher.localhost should work and than if traefik is running with port 80 and 443 bound. That should work on both Windows, Linux and MacOS.

The problem on Linux becomes that port 80 and 443 is not able to be used for traefik ingress. You would need to use sudo sysctl net.ipv4.ip_unprivileged_port_start=80 or figure out the correct sudo setcap cap_net_bind_service=+ep /path/to/rancher-desktop

I am currently not running on Linux so I don't know the exact path to rancher-desktop or potentially that would be needed for the k3s binary?

See #1668

[thehejik]

@jetersen thanks, just succesfully verified that:

sudo sysctl -w net.ipv4.ip_unprivileged_port_start=443

works on RD for Linux - user can access traefik ingress and services behind it directly by using https://localhost (without nodeport resolution)

[jandubois]

Nowadays anything .localhost should route to 127.0.0.1

I believe this is just a browser feature (internal redirect), so it will work in Chrome and Firefox, but not with Safari.

So it is not implemented via DNS and will not work with curl or wget, or any programs trying to access a service programatically.

[mattupstate]

I believe this is just a browser feature (internal redirect), so it will work in Chrome and Firefox, but not with Safari.

systemd-resolved will resolve domains with TLD .localhost to loopback device. For example, this is on my Ubuntu 21.10 desktop with no other configuration around DNS:

$ curl -v hello.localhost
*   Trying ::1:80...
* connect to ::1 port 80 failed: Connection refused
*   Trying 127.0.0.1:80...
* Connected to hello.localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: hello.localhost
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Date: Wed, 04 May 2022 19:22:00 GMT
< Content-Type: text/html
< Content-Length: 146
< Connection: keep-alive
< 
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host hello.localhost left intact

[jandubois]

systemd-resolved will resolve domains with TLD .localhost to loopback device.

That's good to know, thank you!

But the original claim was:

That should work on both Windows, Linux and MacOS.

So this seems to be true if you use Linux with systemd, but otherwise you still depend on the browser to support it. It is not universal: It does not work on macOS, Windows, or Linux without systemd.

[simonflood]

@jandubois if I can qualify your last comment

"So this seems to be true if you use Linux with systemd and the systemd-resolved service is enabled" as it's not by default on openSUSE Leap 15.3 which I'm currently using - the service is there, just wasn't enabled (and I didn't disable it).

[jandubois]

if I can qualify your last comment

@simonflood I have no idea what you are trying to say. If you don't use systemd-resolved, then *.localhost will not resolve on Linux either. That's essentially what I was saying all along; support for this feature is far from universal:

jan@localhost:~> grep PRETTY /etc/os-release 
PRETTY_NAME="openSUSE Leap 15.2"
jan@localhost:~> ping foo.localhost
ping: foo.localhost: Temporary failure in name resolution

[simonflood]

@simonflood I have no idea what you are trying to say. If you don't use systemd-resolved, then *.localhost will not resolve on Linux either. That's essentially what I was saying all along; support for this feature is far from universal:

Thanks, I was just noting that Linux with systemd doesn't necessarily give you this as you also need systemd-resolved enabled.

[gaktive]

Of note that is that Epinio has documented the workaround cited earlier. Maybe we need to do the same in the Rancher Desktop docs.

URL: https://docs.epinio.io/howtos/install_epinio_on_rancher_desktop#rancher-desktop-prerequisites PR: https://github.com/epinio/docs/pull/127

[JohnGalt1717]

So, I've read this, but I'm still at a loss (the iptables command doesn't work even after taking the privilage away) as to how to get 80 and 443 working on 0.0.0.0 (or even 127.0.0.0)

Does anyone have any known good steps to make this work? (specifically I need 0.0.0.0 access to rancher desktop so that android/ios emulators etc can access the cluster services)

[jandubois]

The commands at the bottom of that Epinio page should work for getting the port forwarded to 127.0.0.1, e.g.

sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80

To get the port bound to an external interface, you could maybe try to setup a tap device on the host, add a portForwards section to an override.yaml file, and then forward from the tap device. I haven't tried this myself, so I don't know if it will work (I think it should, but I don't know), and I have no time to look into this any time soon.

@Nino-K Do you have any better ideas?

Using iptables does not work with localhost addresses, as those packets are delivered directly inside the kernel and are never seen by the filter, so they can't be redirected.

[Nino-K]

@JohnGalt1717 as a short-term solution you would need to either run the iptables rules or if you are on a system with either root access or capabilities you can run what is mentioned above:

The problem on Linux becomes that port 80 and 443 is not able to be used for traefik ingress. You would need to use sudo sysctl net.ipv4.ip_unprivileged_port_start=80 or figure out the correct sudo setcap cap_net_bind_service=+ep /path/to/rancher-desktop

However, please note if you have a kernel with capabilities, be aware when running:

sudo setcap cap_net_bind_service=+ep /path/to/rancher-desktop

the above set capability command basically allows any nonprivileged user to run the rancher desktop on privileged ports so you need to be mindful of this.

I would recommend using the iptables solution, it should work fine, here are the steps that you may need to take:

• Define 2 sets of NAT rules one for PREROUTING (for altering packets as soon as they come in), and one for OUTPUT (for altering locally-generated packets before routing) like below.

root@myhost # $nodeport=$(kubectl get svc -n kube-system traefik -o jsonpath='{.spec.ports[1].nodePort}')
root@myhost # iptables -t nat -A PREROUTING -d <ip> -p tcp --dport 80 -m addrtype --dst-type LOCAL -j DNAT --to-destination <ip>:$nodeport
root@myhost # iptables -t nat -A OUTPUT     -d <ip> -p tcp --dport 80 -m addrtype --dst-type LOCAL -j DNAT --to-destination <ip>:$nodeport

Please note, set the <ip> to 0.0.0.0 if you want to have traefik on all IP addresses otherwise leave it at whatever IP you prefer for your work.

Another important thing to note is if the rules above are not working for you, that might be because it also requires the kernel to enable (take a look here for more details):

sysctl -w net.ipv4.conf.all.route_localnet=1

if with the kernel option enable you are still facing issues, that could be because the rules defined above are sitting below a REJECT or DROP rule associated with the NAT, you would need to look at your iptables rule to figure that out.

@jandubois as a short-term and temporary solution I cannot think of any other solution but what is mentioned above. The more appropriate long-term would be for RD to start up a bridge interface (TAP device) to which we can associate all our port binding activities.