k3d
k3d copied to clipboard
k3d-io
K3D on Ubuntu with Metallb and Unifi USG via BGP
Hi, I have set up k3d on a ubuntu 22 LTS server and trying to get Metallb to work with Unifi USG to get a working Internal (within LAN) Load Balancer IP with a different subnet.
I have followed the docs from Metallb to get BGP working (https://metallb.universe.tf/configuration/) and have added the CRDs below:
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
namespace: metallb
name: unifi-usg-addr-pool
labels:
router: unifi-usg
dhcp: unifi-usg
spec:
addresses:
- 10.1.1.6-10.1.1.250
----
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
name: unfi-usg-bgp-adv
namespace: metallb
----
apiVersion: metallb.io/v1beta1
kind: BGPPeer
metadata:
name: unfi-usg-bgp-peer
namespace: metallb
spec:
myASN: 64512
peerASN: 64512
peerAddress: 10.0.0.1
My current subnet is 10.0.0.0/24 with the router-ip being 10.0.0.1. I created the k3d cluster with the following flags on my Ubuntu server (10.0.0.26 IP address):
k3d cluster create homelab --api-port 6550 --agents 1 --k3s-arg "--disable=traefik@server:*" --k3s-arg "--disable=servicelb@server:*" --k3s-arg "--kube-proxy-arg=--ipvs-strict-arp=true@server:*" --no-lb --wait
My plan is to use ingress-nginx as the Ingress Controller and have that get a LoadBalancer service with an IP address with routing from the Unifi USG. I created a test-nginx service as a first step and can see that is getting an IP from the metallb ip address pool:
nginx LoadBalancer 10.43.20.20 10.1.1.6 80:30355/TCP 60m
I also set up bgp routing in unifi usg as below:
bgp 64512 {
neighbor 10.0.0.26 {
remote-as 64512
}
parameters {
router-id 10.0.0.1
}
}
But in the show ip bgp output, I see that it is directing hops to 172.19.0.2:
admin@ubnt:~$ show ip bgp
BGP table version is 0, local router ID is 10.0.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*>i10.1.1.6/32 172.19.0.2 0 0 i
Total number of prefixes 1
That 172.19.0.2 ip address is for the docker container that runs the agent of k3d: "Hostname": "k3d-homelab-agent-0"
"Gateway": "172.19.0.1",
"IPAddress": "172.19.0.2",
The current routing and configuration is not working as I am unable to get a response back from 10.1.1.6 (assigned LB IP). Does anyone have a similar experience with some resolution and could point me in the right direction?
It appears the hop is going directly to the agent which kind of makes sense since its the one actually handling communication with pods/services but not really since how it would directly communicate with the docker container without going through the host/Ubuntu server? I believe there might need to be some type of bridge network between the two (host and container) but I'm unsure what exactly to try next.
Any help appreciated, thanks!
Relevant Versions:
// k3d
k3d version v5.4.6
k3s version v1.24.4-k3s1 (default)
//metallb via helm chart
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
metallb metallb 1 2022-09-21 01:06:43.8366683 -0400 EDT deployed metallb-0.13.5 v0.13.5
// kube versions
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"5835544ca568b757a8ecae5c153f317e5736700e", GitTreeState:"clean", BuildDate:"2022-09-21T14:33:49Z", GoVersion:"go1.19.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.4+k3s1", GitCommit:"c3f830e9b9ed8a4d9d0e2aa663b4591b923a296e", GitTreeState:"clean", BuildDate:"2022-08-25T03:45:26Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
One thing I did notice that I'm not too sure about but looks wrong is when I look into the unifi USG config, I can see that the subnet I allocated to Metallb is showing up in the ip routes:
admin@ubnt:~$ ip route
default via 192.168.1.254 dev eth0 proto zebra
10.0.0.0/24 dev eth1 proto kernel scope link src 10.0.0.1
10.1.1.6 via 192.168.1.254 dev eth0 proto zebra
10.1.1.7 via 192.168.1.254 dev eth0 proto zebra
127.0.0.0/8 dev lo proto kernel scope link src 127.0.0.1
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.64
We can see the 10.1.1.6 IP address allocated to the nginx test server in the output and it shows as using eth1 interface. My current subnet of 10.0.0.0/24 also shows using eth0 interface. But when I look at network interfaces:
admin@ubnt:~$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
eth0 192.168.1.64/24 u/u WAN
2600:1700:28a0:bd30::48/128
eth1 10.0.0.1/24 u/u LAN
eth2 - A/D
lo 127.0.0.1/8 u/u
::1/128
10.0.0.0/24 using eth1 interface looks correct since this is my LAN network. But seeing the 10.1.1.6 IP address using the eth0 network seems wrong since it's purpose is for WAN and would be public facing. Shouldn't it being using the same interface, eth1, as my LAN subnet?
So I ended up going with k3s instead due to the issue with the next-hop showing as the docker container instead of the actual host. From my limited knowledge that is something that won't work unless there is some type of bridge network or tunnel between the container and the host. Personally I'd rather not introduce more supporting services/requirements to get this to work so ended up going with K3s. In the snippet below, the next-hop is now correctly showing up as the k3s server internal IP address:
admin@ubnt:~$ show ip bgp
BGP table version is 0, local router ID is 10.0.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, R Removed
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*>i10.1.1.6/32 10.0.0.26 0 0 i
I also don't see the issue with the Metallb subnet using the eth0/WAN interface in USG after switching to k3s:
admin@ubnt:~$ ip route
default via 192.168.1.254 dev eth0 proto zebra
10.0.0.0/24 dev eth1 proto kernel scope link src 10.0.0.1
10.1.1.6 via 10.0.0.26 dev eth1 proto zebra
127.0.0.0/8 dev lo proto kernel scope link src 127.0.0.1
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.64
It is now properly using the eth1/LAN interface as I thought it should. I am able to hit the nginx-test service with the allocated IP address of the different subnet (the one assigned to Metallb):
curl -X get http://10.1.1.6:80
<html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>
For anyone who is looking to do something similar in the future, here are some additional steps I did to get this to work:
On Unifi USG, enable bgp, example with the neighbor being my k3s node:
configure
set protocols bgp 64512 parameters router-id 10.0.0.1
set protocols bgp 64512 neighbor "10.0.0.26" remote-as 64512
commit
save
If using Ubuntu, to enable cluster access form other LAN hosts, you need to add an allow rule in ufw for ipv6 port 6443 (or your api server port). Note when I installed K3s, it was using/listening on ipv6, at least in my set up so this was required. ufw is enabled by default and you can either allow 6443 ipv6 traffic with a blanket statement or be more specific by specifying your LAN subnets(s). Blanket allow all rule example below but can be more specific:
netstat -tulpn
tcp6 0 0 :::6443 :::* LISTEN 83327/k3s server
ufw allow 6443
ufw status numbered
Status: active
To Action From
-- ------ ----
[ 1] 443/tcp ALLOW IN Anywhere
[ 2] OpenSSH ALLOW IN Anywhere
[ 3] 6443 ALLOW IN Anywhere
[ 4] 443/tcp (v6) ALLOW IN Anywhere (v6)
[ 5] OpenSSH (v6) ALLOW IN Anywhere (v6)
[ 6] 6443 (v6) ALLOW IN Anywhere (v6)
I then could start/install K3s with the following flags:
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable=traefik --disable=servicelb --kube-proxy-arg=--ipvs-strict-arp=true" sh -
According to Metallb docs, you should always specify --disable=servicelb and -kube-proxy-arg=--ipvs-strict-arp flags for this to work. I also disabled Traefik (don't have to, I'm sure it's fine to use) since I am much more familiar with nginx-ingress and prefer to use it for my day to day ingress controller.
I am unsure of what exactly needs to be done to fix this issue with k3d but if anyone ever wants to investigate/resolve it, I ddo hope this helps. The two things I see so far that need be fixed/added (again limited knowledge but learning, so may be wrong):
- Bridge network (Container and host)?
- Metallb subnet using the correct interface in the USG route table
I will leave this ticket up just in case anyone wants to take a look or if it will hopefully provide some help to any users. But if not applicable, please feel free to close.