Ingress + MetalLB not working

[VitorNilson]

Summary

I have installed Microk8s sudo snap install microk8s --classic --channel=1.27.

This automatically enables the addon ingress.

I have enabled MetalLB: microk8s.enable metallb:$(curl ipinfo.io/ip)-$(curl ipinfo.io/ip) and opened the port 80 and 443 on my OCI network.

This is the deployment that I'm using as test:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: nginx
  name: nginx-deploy-main
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx-main
  template:
    metadata:
      labels:
        run: nginx-main
    spec:
      containers:
      - image: nginx
        name: nginx
---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-resource-2
spec:
  ingressClassName: nginx
  rules:
  - host: test.my.addres.net
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-deploy-main
            port:
              number: 80

---
# kubectl get svc ... -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2023-07-15T16:56:53Z"
  labels:
    run: nginx
  name: nginx-deploy-main
  namespace: default
  resourceVersion: "28822"
  uid: f4a701e8-621b-4cda-8059-7e0196e4bfbb
spec:
  clusterIP: 10.152.183.170
  clusterIPs:
  - 10.152.183.170
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx-main
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Addons enabled:

microk8s is running
high-availability: no
  datastore master nodes: 10.0.0.72:19001
  datastore standby nodes: none
addons:
  enabled:
    cert-manager         # (core) Cloud native certificate management
    dns                  # (core) CoreDNS
    ha-cluster           # (core) Configure high availability on the current node
    helm                 # (core) Helm - the package manager for Kubernetes
    helm3                # (core) Helm 3 - the package manager for Kubernetes
    ingress              # (core) Ingress controller for external access
    metallb              # (core) Loadbalancer for your Kubernetes cluster
  disabled:
    community            # (core) The community addons repository
    dashboard            # (core) The Kubernetes dashboard
    host-access          # (core) Allow Pods connecting to Host services smoothly
    hostpath-storage     # (core) Storage class; allocates storage from host directory
    kube-ovn             # (core) An advanced network fabric for Kubernetes
    mayastor             # (core) OpenEBS MayaStor
    metrics-server       # (core) K8s Metrics Server for API access to service metrics
    minio                # (core) MinIO object storage
    observability        # (core) A lightweight observability stack for logs, traces and metrics
    prometheus           # (core) Prometheus operator for monitoring and logging
    rbac                 # (core) Role-Based Access Control for authorisation
    registry             # (core) Private image registry exposed on localhost:32000
    storage              # (core) Alias to hostpath-storage add-on, deprecated

Ingress Namespace:

ubuntu@oci-instance:~$ kubectl get all -n ingress
NAME                                          READY   STATUS    RESTARTS   AGE
pod/nginx-ingress-microk8s-controller-fzj5h   1/1     Running   0          169m

NAME                                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/nginx-ingress-microk8s-controller   1         1         1       1            1           <none>          169m

Kube-System namespace:

ubuntu@oci-instance:~$ kubectl get all -n kube-system
NAME                                           READY   STATUS    RESTARTS       AGE
pod/coredns-7745f9f87f-jfkjk                   1/1     Running   4 (4h2m ago)   7h18m
pod/calico-kube-controllers-6c99c8747f-pt9d9   1/1     Running   4 (4h2m ago)   7h18m
pod/calico-node-dgzkt                          1/1     Running   3 (4h2m ago)   6h45m

NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns   ClusterIP   10.152.183.10   <none>        53/UDP,53/TCP,9153/TCP   7h18m

NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/calico-node   1         1         1       1            1           kubernetes.io/os=linux   7h18m

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/calico-kube-controllers   1/1     1            1           7h18m
deployment.apps/coredns                   1/1     1            1           7h18m

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/calico-kube-controllers-6c99c8747f   1         1         1       7h18m
replicaset.apps/coredns-7745f9f87f                   1         1         1       7h18m

metallb-system Namespace:

ubuntu@oci-instance:~$ kubectl get all -n metallb-system
NAME                              READY   STATUS    RESTARTS   AGE
pod/controller-8467d88d69-bhj2p   1/1     Running   0          179m
pod/speaker-nwrvf                 1/1     Running   0          179m

NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/webhook-service   ClusterIP   10.152.183.36   <none>        443/TCP   179m

NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/speaker   1         1         1       1            1           kubernetes.io/os=linux   179m

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/controller   1/1     1            1           179m

NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/controller-8467d88d69   1         1         1       179m

If I try to test:

curl -H "Host: test.my.addres.net" http://144.11.111.111 
curl: (7) Failed to connect to 144.11.111.111 port 80 after 22 ms: Connection refused

If I convert this ingress to NodePort, open the port on OCI it works without any problem.

What am I doing wrong?

[berkayoz]

Hey @VitorNilson,

Seems like there are some missing setup steps. I'd suggest checking out our MetalLB docs, which explains how the setup should work.

Thanks.

[neoaggelos]

@VitorNilson it looks like you deploy a Service of type ClusterIP, you should use LoadBalancer instead for MetalLB to assign an external IP address to it.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.