k3d
k3d copied to clipboard
k3d-io
[ENHANCEMENT] Creative solution for port binding issues?
Right now you have to do -p to bind any port you'd wish to access k8s. On linux this isn't much of an issue because you can just access the k3d container IP directly but on docker-for-mac and windows this isn't possible. I'm wondering if we could get creative here. What we could do is deploy a controller that watches hosts ports of pods and then dynamically create docker containers that just forward that one port. For example....
If I can create a service LB in k3s that exposes port 80, what we do is actually create a pod that uses host port 80 and forwards all request to the service LB's cluster IP. We would see that a pod is using host port 80 and deploy a docker container with -p 80:80. In that docker container we would then create an iptables rule that forwards all traffic coming in on 80 and forwards to the k3s_default container. Then VPNKit on docker-for-mac, docker-for-windows will create a forwarding tunnel from the desktop to the new docker container we just created.
The only issue I see with this approach is if you have multiple k3s clusters running. We could do something then like "port offset". So one cluster gets an offset of 0 so port 80 and 443 will be bound to 80 and 443. Then a second cluster would get an offset like 10000 so ports get bound to 10080 and 10443.
WDTY?
Instead of dynamically creating docker container, another thing we can do is run the port-watcher in host network, it will watch ports and forward them as required using a proxy server.
About multiple k3s clusters the port offset option seems good.
@zeerorg How would the port-watcher work? The basic issue is that on docker-for-mac the VM IP is not routable to the desktop (outside the VM). So creating docker containers is a hack to get VPNKit to create tunnels to the host.
One idea is to query the kubernetes api for all "HostPort" services every fixed time interval. and then expose those.
Another idea is to smartly use kubectl port-forward to just skip creating docker containers and instead run a k3d service in the background that handles redirects.
We could probably do something really intelligent and do k3d port-forward that dynamically binds and forwards ports. It wouldn't be too hard to write a controller to do that.
I'll try a sample implementation of this, just do k3d port-forward and for all HostPort and NodePort binds run kubectl port-forward.
Running a different docker container for every port seems like it could go out of hand if we have lots of ports. Do you have any adv/disadv for both of these?
Immediate adv that second implementation has is that it could do udp port-forward but it will be harder to implement.
There's also this directory in the moby VM which directly manipulates the VPNKit port mapping, but it's not clear if there's any way to get to it from outside (-v /var/vpnkit/port:/port tries to find that on the exported list of the Mac):
# cat /var/vpnkit/port/README
Directory of active Instances
-----------------------------
Every active Instance is represented by a file. To shut down an Instance,
remove the file.
To request an additional Instance, make a directory with a unique name,
then open the special file `ctl` inside, and `write` a single string of the
following form:
tcp:<local IP>:<local port>:tcp:<remote IP>:<remote port>
udp:<local IP>:<local port>:udp:<remote IP>:<remote port>
unix:<base64-encoded local path>:unix:<base64-encoded remote path>
Immediately read the file contents and check whether it says:
- `OK <instance details>`: this means the Instance has been configured and
the details returned to you. For some instance types the server might modify
the request slightly, for example by choosing a local port number or
temporary path.
- `ERROR some error message`: this means the Instance creation has failed,
perhaps some needed resource is still in use.
The directory will be deleted and replaced with a file of the same name.
vpnkit port mapping certainly seems the right thing to do, I'll try to scrounge docker windows and mac source code to see how vpnkit is being used.
Awesome, automatic port binding would greatly increase the usability.
I don't know where the /var/vpnkit/port mount comes from, but on the mac host there is few unix sockets in ~/Library/Containers/com.docker.docker/Data/
srwxr-xr-x 1 ash staff 0 3 Apr 16:19 vpnkit.diag.sock
srwxr-xr-x 1 ash staff 0 3 Apr 16:19 vpnkit.eth.sock
srwxr-xr-x 1 ash staff 0 3 Apr 16:19 vpnkit.pcap.sock
srwxr-xr-x 1 ash staff 0 3 Apr 16:19 vpnkit.port.sock
And the code in VPNKit that speakes to it: https://github.com/moby/vpnkit/blob/2b3920dab1da4c5cddeb94f4feb31a18700fefa9/go/pkg/vpnkit/port.go (and other files in that folder)
Any thoughts on this https://github.com/rancher/k3d/issues/89 ? @zeerorg could you find anything about using vpnkit?
@ibuildthecloud Why isn't it possible to access the docker container IP on macos?
@ibuildthecloud Why isn't it possible to access the docker container IP on macos?
Because on MacOS, docker is running inside a VM, so there would have to be a route host -> Docker VM -> container
Because on MacOS, docker is running inside a VM, so there would have to be a route host -> Docker VM -> container
Ahh and does it mean that if I run docker container inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' CONTAINER_ID, the resulting IP address only makes sense inside the VM?
Because on MacOS, docker is running inside a VM, so there would have to be a route host -> Docker VM -> container
Ahh and does it mean that if I run
docker container inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' CONTAINER_ID, the resulting IP address only makes sense inside the VM?
Yep, to my knowledge that's how it is.