loft-sh
Load Balancer IP is not synced to host after editing `spec.loadBalancerIP` from vcluster
What happened?
I'm trying to create and edit a service from the cluster with LoadBalancer type. The host is using Calico and Metallb for the network. Here is the Metallb configmap:
apiVersion: v1
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 192.168.56.10-192.168.56.20
kind: ConfigMap
metadata:
name: config
namespace: metallb-system
Helm values:
{
"api": {
"image": "k8s.gcr.io/kube-apiserver:v1.23.1",
"replicas": 1,
"resources": {
"limits": {
"cpu": "500m",
"memory": "512Mi"
},
"requests": {
"cpu": "200m",
"memory": "128Mi"
}
}
},
"etcd": {
"image": "k8s.gcr.io/etcd:3.5.1-0",
"storage": {
"size": "5.0Gi"
},
"replicas": 1,
"resources": {
"limits": {
"cpu": "500m",
"memory": "512Mi"
},
"requests": {
"cpu": "100m",
"memory": "128Mi"
}
}
},
"sync": {
"storageclasses": {
"enabled": true
},
"networkpolicies": {
"enabled": true
},
"priorityclasses": {
"enabled": true
},
"serviceaccounts": {
"enabled": true
},
"volumesnapshots": {
"enabled": true
},
"persistentvolumes": {
"enabled": true
},
"poddisruptionbudgets": {
"enabled": true
}
},
"syncer": {
"replicas": 1,
"resources": {
"limits": {
"cpu": "500m",
"memory": "512Mi"
},
"requests": {
"cpu": "100m",
"memory": "128Mi"
}
}
},
"service": {
"type": "ClusterIP"
},
"enableHA": false,
"controller": {
"image": "k8s.gcr.io/kube-controller-manager:v1.23.1",
"replicas": 1,
"resources": {
"limits": {
"cpu": "500m",
"memory": "512Mi"
},
"requests": {
"cpu": "200m",
"memory": "128Mi"
}
}
},
"serviceCIDR": "10.1.0.0/16"
}
The service is successfully created and the ip is assigned at the first time. But, if I tried to edit the service and change the IP from vcluster, it's not getting synced with the host.
$ kubectl expose pod/test-pod --type LoadBalancer --port 80 --load-balancer-ip 192.168.56.10
service/test-pod exposed
Here I can see the service is created in the vcluster
$ kubectl --kubeconfig kubeconfig get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.1.169.191 <none> 443/TCP 1h
test-pod LoadBalancer 10.1.139.169 192.168.56.10 80:31911/TCP 67s
and also host
$ kubectl -n k8s-7157e2ee-afd5-4658-9086-d00bc4f74eee get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns-x-kube-system-x-mycluster ClusterIP 10.1.204.189 <none> 53/UDP,53/TCP,9153/TCP 1h
mycluster ClusterIP 10.1.169.191 <none> 443/TCP 1h
mycluster-api ClusterIP 10.1.244.33 <none> 443/TCP 1h
mycluster-etcd ClusterIP 10.1.63.125 <none> 2379/TCP,2380/TCP 1h
mycluster-etcd-headless ClusterIP None <none> 2379/TCP,2380/TCP 1h
mycluster-node-awid3 ClusterIP 10.1.149.199 <none> 10250/TCP 1h
test-pod-x-default-x-mycluster LoadBalancer 10.1.139.169 192.168.56.10 80:31911/TCP 2m53s
But, after editing spec.loadBalancerIP (using kubectl edit or kubectl apply or kubectl patch) from the vcluster to 192.168.56.11, The IP is not getting changed both in the cluster and host.
$ kubectl --kubeconfig kubeconfig get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.1.169.191 <none> 443/TCP 1h
test-pod LoadBalancer 10.1.139.169 192.168.56.10 80:31911/TCP 5m7s
host:
$ kubectl -n k8s-7157e2ee-afd5-4658-9086-d00bc4f74eee get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns-x-kube-system-x-mycluster ClusterIP 10.1.204.189 <none> 53/UDP,53/TCP,9153/TCP 1h
mycluster ClusterIP 10.1.169.191 <none> 443/TCP 1h
mycluster-api ClusterIP 10.1.244.33 <none> 443/TCP 1h
mycluster-etcd ClusterIP 10.1.63.125 <none> 2379/TCP,2380/TCP 1h
mycluster-etcd-headless ClusterIP None <none> 2379/TCP,2380/TCP 1h
mycluster-node-awid3 ClusterIP 10.1.149.199 <none> 10250/TCP 1h
test-pod-x-default-x-mycluster LoadBalancer 10.1.139.169 192.168.56.10 80:31911/TCP 7m5s
This is what I got from kubectl --kubeconfig kubeconfig get events
...
6m6s Normal IPAllocated service/test-pod Assigned IP ["192.168.56.10"]
6m6s Normal nodeAssigned service/test-pod announcing from node "awid"
19s Normal LoadbalancerIP service/test-pod 192.168.56.10 -> 192.168.56.11
19s Normal LoadbalancerIP service/test-pod 192.168.56.11 -> 192.168.56.10
If I change the value from the host, it works as expected:
$ kubectl --kubeconfig kubeconfig get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.1.169.191 <none> 443/TCP 1h
test-pod LoadBalancer 10.1.139.169 192.168.56.11 80:31911/TCP 9m50s
host:
$ kubectl -n k8s-7157e2ee-afd5-4658-9086-d00bc4f74eee get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns-x-kube-system-x-mycluster ClusterIP 10.1.204.189 <none> 53/UDP,53/TCP,9153/TCP 1h
mycluster ClusterIP 10.1.169.191 <none> 443/TCP 1h
mycluster-api ClusterIP 10.1.244.33 <none> 443/TCP 1h
mycluster-etcd ClusterIP 10.1.63.125 <none> 2379/TCP,2380/TCP 1h
mycluster-etcd-headless ClusterIP None <none> 2379/TCP,2380/TCP 1h
mycluster-node-awid3 ClusterIP 10.1.149.199 <none> 10250/TCP 1h
test-pod-x-default-x-mycluster LoadBalancer 10.1.139.169 192.168.56.11 80:31911/TCP 11m
What did you expect to happen?
The load balancer IP should be changed as specified if the value is edited from vcluster.
How can we reproduce it (as minimally and precisely as possible)?
- Create a pod
- Expose the pod and specify the load balancer IP
- Edit the load balancer IP from vcluster
- Observe the service both from vcluster and the host
Anything else we need to know?
The host using Calico and Metallb
Host cluster Kubernetes version
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:25:17Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4+k3s1", GitCommit:"43b1cb48200d8f6af85c16ed944d68fcc96b6506", GitTreeState:"clean", BuildDate:"2022-02-24T22:38:17Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Host cluster Kubernetes distribution
kubeadm and k3s
vlcuster version
$ vcluster --version
vcluster version 0.7.1
Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)
k8s
OS and Arch
OS: Linux Ubuntu 20.04.2 LTS
Arch: x86
@akbarkn thanks for creating this issue! vcluster currently only handles the case that the host cluster is setting the load balancer ip, so setting it inside the vcluster only works on creation as values are copied to the host cluster for creation. But we might also want to allow setting it from inside the vcluster.
@FabianKramm Thanks for the answer. Nice to hear that you will handle the case in the future.
@FabianKramm I am able to replicate the same, can you please assign this bug to work on!
@sandeep540 Are you still working on this issue? If you got blocked and need help, please don't hesitate to reach out to us in vcluster channel in our Slack.
Hi @matskiv , No I am currently not working on this! you can reassign it to someone