"Failed to ensure portal" running kind cluster

Open siredmar opened this issue 3 years ago • 11 comments

What happened: I'm trying to run edgemesh on a kind cluster. I have one control-plane node and two workers. I want default-worker to act as a relay server.

$ kubectl get nodes -o wide        
NAME                    STATUS   ROLES                  AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION    CONTAINER-RUNTIME
default-control-plane   Ready    control-plane,master   3d9h   v1.22.9   10.5.0.2      <none>        Ubuntu 21.10   5.17.0-1016-oem   containerd://1.6.4
default-worker          Ready    <none>                 3d9h   v1.22.9   10.5.0.4      <none>        Ubuntu 21.10   5.17.0-1016-oem   containerd://1.6.4
default-worker2         Ready    <none>                 3d9h   v1.22.9   10.5.0.3      <none>        Ubuntu 21.10   5.17.0-1016-oem   containerd://1.6.4

This is my values.yaml

agent:
  image: kubeedge/edgemesh-agent:latest
  affinity: {}
  nodeSelector: {}
  tolerations: {}
  resources:
    limits:
      cpu: 1
      memory: 256Mi
    requests:
      cpu: 0.5
      memory: 128Mi
  psk: dAc+kaXv1dLeDNB4JR79LwBQCwvBx6k6t5UtinL6OiU=
  relayNodes:
    - nodeName: default-worker
      advertiseAddress:
        - 10.5.0.4
  modules:
    edgeProxy:
      enable: true
    edgeTunnel:
      enable: true

When i deploy the helm chart using my values.yaml i get two pods on my worker nodes

$ kubectl get pods -o wide
NAME                                READY   STATUS    RESTARTS      AGE     IP            NODE                    NOMINATED NODE   READINESS GATES
cloudcore-55f44b557f-zsf9f          2/2     Running   2 (14m ago)   3d9h    10.5.0.4      default-worker          <none>           <none>
edgemesh-agent-2fsjb                1/1     Running   0             6s      10.5.0.3      default-worker2         <none>           <none>
edgemesh-agent-jgwk9                1/1     Running   0             6s      10.5.0.4      default-worker          <none>           <none>
iptables-manager-fwbvb              1/1     Running   0             3d9h    10.5.0.2      default-control-plane   <none>           <none>

However the logs show, that there is some errors

$ kubectl logs edgemesh-agent-2fsjb    
I1010 06:38:34.199247       1 server.go:55] Version: v1.12.0-dirty
I1010 06:38:34.199286       1 server.go:89] [1] Prepare agent to run
I1010 06:38:34.199419       1 netif.go:96] bridge device edgemesh0 already exists
I1010 06:38:34.199473       1 server.go:93] edgemesh-agent running on CloudMode
I1010 06:38:34.199481       1 server.go:96] [2] New clients
W1010 06:38:34.199492       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1010 06:38:34.199903       1 server.go:103] [3] Register beehive modules
W1010 06:38:34.199915       1 module.go:37] Module EdgeDNS is disabled, do not register
I1010 06:38:34.200252       1 server.go:66] Using userspace Proxier.
I1010 06:38:34.292646       1 module.go:34] Module EdgeProxy registered successfully
I1010 06:38:34.362580       1 module.go:159] I'm {12D3KooWNLAPNyViyXgHyAoTtvPc9D8fW3R5RJCAF8iiwpALCUQY: [/ip4/10.5.0.3/tcp/20006 /ip4/127.0.0.1/tcp/20006]}
I1010 06:38:34.362659       1 module.go:181] Bootstrapping the DHT
I1010 06:38:34.362689       1 tunnel.go:387] [Bootstrap] bootstrapping to 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw
E1010 06:38:34.363161       1 tunnel.go:391] [Bootstrap] failed to bootstrap with {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/10.5.0.4/tcp/20006]}: failed to dial 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw:
  * [/ip4/10.5.0.4/tcp/20006] dial tcp4 10.5.0.4:20006: connect: connection refused
E1010 06:38:34.363282       1 tunnel.go:402] [Bootstrap] Not all bootstrapDail connected, continue bootstrapDail...
I1010 06:38:44.364467       1 tunnel.go:387] [Bootstrap] bootstrapping to 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw
I1010 06:38:44.372216       1 tunnel.go:397] [Bootstrap] success bootstrapped with {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/10.5.0.4/tcp/20006]}
I1010 06:38:44.373418       1 tunnel.go:63] Starting MDNS discovery service
I1010 06:38:44.373442       1 tunnel.go:76] Starting DHT discovery service
I1010 06:38:44.373510       1 module.go:34] Module EdgeTunnel registered successfully
I1010 06:38:44.373524       1 server.go:109] [4] Start all modules
I1010 06:38:44.373601       1 tunnel.go:447] Starting relay finder
I1010 06:38:44.373624       1 core.go:24] Starting module EdgeProxy
I1010 06:38:44.373692       1 core.go:24] Starting module EdgeTunnel
I1010 06:38:44.373950       1 config.go:317] "Starting service config controller"
I1010 06:38:44.374019       1 shared_informer.go:240] Waiting for caches to sync for service config
I1010 06:38:44.373959       1 config.go:135] "Starting endpoints config controller"
I1010 06:38:44.375185       1 shared_informer.go:240] Waiting for caches to sync for endpoints config
I1010 06:38:44.375746       1 loadbalancer.go:239] "Starting loadBalancer destinationRule controller"
I1010 06:38:44.376100       1 shared_informer.go:240] Waiting for caches to sync for loadBalancer destinationRule
I1010 06:38:44.382517       1 tunnel.go:175] Discovery service got a new stream from {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/10.5.0.4/tcp/20006]}
I1010 06:38:44.382894       1 tunnel.go:204] [MDNS] Discovery from default-worker : {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/10.5.0.4/tcp/20006]}
I1010 06:38:44.383297       1 tunnel.go:118] [MDNS] Discovery found peer: {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/10.5.0.4/tcp/20006 /ip4/127.0.0.1/tcp/20006]}
I1010 06:38:44.383478       1 tunnel.go:130] [MDNS] New stream between peer {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/127.0.0.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/169.254.96.16/tcp/20006 /ip4/10.5.0.4/tcp/20006]} success
I1010 06:38:44.455824       1 tunnel.go:166] [MDNS] Discovery to default-worker : {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/127.0.0.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/169.254.96.16/tcp/20006 /ip4/10.5.0.4/tcp/20006]}
I1010 06:38:44.475928       1 shared_informer.go:247] Caches are synced for endpoints config 
I1010 06:38:44.475991       1 shared_informer.go:247] Caches are synced for service config 
I1010 06:38:44.476404       1 shared_informer.go:247] Caches are synced for loadBalancer destinationRule 
E1010 06:38:44.668305       1 proxier.go:552] "Failed to open portal" err="can't open node port for <nil>:30972/TCP: listen tcp :30972: bind: address already in use" serviceName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1010 06:38:44.688464       1 proxier.go:552] "Failed to open portal" err="can't open node port for <nil>:31163/TCP: listen tcp :31163: bind: address already in use" serviceName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1010 06:38:44.786637       1 proxier.go:552] "Failed to open portal" err="can't open node port for <nil>:30615/TCP: listen tcp :30615: bind: address already in use" serviceName="metallb-system/nginx"
E1010 06:38:44.824920       1 proxier.go:552] "Failed to open portal" err="can't open node port for <nil>:30550/TCP: listen tcp :30550: bind: address already in use" serviceName="vault/vault:vault"
E1010 06:38:44.845249       1 proxier.go:552] "Failed to open portal" err="can't open node port for <nil>:30738/TCP: listen tcp :30738: bind: address already in use" serviceName="vault/vault:vault-cluster"
E1010 06:38:44.889235       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30550/TCP: listen tcp :30550: bind: address already in use" servicePortName="vault/vault:vault"
E1010 06:38:44.964074       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30738/TCP: listen tcp :30738: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1010 06:38:45.029171       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30972/TCP: listen tcp :30972: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1010 06:38:45.044993       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31163/TCP: listen tcp :31163: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1010 06:38:45.077913       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30615/TCP: listen tcp :30615: bind: address already in use" servicePortName="metallb-system/nginx"
E1010 06:39:14.443733       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30738/TCP: listen tcp :30738: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1010 06:39:14.476872       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30972/TCP: listen tcp :30972: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1010 06:39:14.490054       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31163/TCP: listen tcp :31163: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1010 06:39:14.504344       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30615/TCP: listen tcp :30615: bind: address already in use" servicePortName="metallb-system/nginx"
E1010 06:39:14.537277       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30550/TCP: listen tcp :30550: bind: address already in use" servicePortName="vault/vault:vault
...

and the other pod

$ kubectl  logs edgemesh-agent-jgwk9 
I1010 06:38:34.175432       1 server.go:55] Version: v1.12.0-dirty
I1010 06:38:34.175470       1 server.go:89] [1] Prepare agent to run
I1010 06:38:34.175594       1 netif.go:96] bridge device edgemesh0 already exists
I1010 06:38:34.175632       1 server.go:93] edgemesh-agent running on CloudMode
I1010 06:38:34.175645       1 server.go:96] [2] New clients
W1010 06:38:34.175656       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1010 06:38:34.176073       1 server.go:103] [3] Register beehive modules
W1010 06:38:34.176085       1 module.go:37] Module EdgeDNS is disabled, do not register
I1010 06:38:34.176347       1 server.go:66] Using userspace Proxier.
I1010 06:38:34.360353       1 module.go:34] Module EdgeProxy registered successfully
I1010 06:38:34.364760       1 module.go:159] I'm {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/10.5.0.4/tcp/20006 /ip4/127.0.0.1/tcp/20006 /ip4/10.5.0.4/tcp/20006]}
I1010 06:38:34.364852       1 module.go:168] Run as a relay node
I1010 06:38:34.364937       1 module.go:181] Bootstrapping the DHT
I1010 06:38:34.366393       1 tunnel.go:63] Starting MDNS discovery service
I1010 06:38:34.366424       1 tunnel.go:76] Starting DHT discovery service
I1010 06:38:34.366470       1 module.go:34] Module EdgeTunnel registered successfully
I1010 06:38:34.366488       1 server.go:109] [4] Start all modules
I1010 06:38:34.366559       1 tunnel.go:447] Starting relay finder
I1010 06:38:34.366582       1 core.go:24] Starting module EdgeProxy
I1010 06:38:34.366621       1 core.go:24] Starting module EdgeTunnel
I1010 06:38:34.366980       1 config.go:135] "Starting endpoints config controller"
I1010 06:38:34.367093       1 shared_informer.go:240] Waiting for caches to sync for endpoints config
I1010 06:38:34.366984       1 config.go:317] "Starting service config controller"
I1010 06:38:34.367164       1 shared_informer.go:240] Waiting for caches to sync for service config
I1010 06:38:34.367200       1 loadbalancer.go:239] "Starting loadBalancer destinationRule controller"
I1010 06:38:34.367225       1 shared_informer.go:240] Waiting for caches to sync for loadBalancer destinationRule
I1010 06:38:34.468202       1 shared_informer.go:247] Caches are synced for loadBalancer destinationRule 
I1010 06:38:34.468234       1 shared_informer.go:247] Caches are synced for service config 
I1010 06:38:34.468246       1 shared_informer.go:247] Caches are synced for endpoints config 
E1010 06:38:34.599795       1 proxier.go:552] "Failed to open portal" err="can't open node port for <nil>:30972/TCP: listen tcp :30972: bind: address already in use" serviceName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1010 06:38:34.678098       1 proxier.go:552] "Failed to open portal" err="can't open node port for <nil>:31163/TCP: listen tcp :31163: bind: address already in use" serviceName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1010 06:38:34.885149       1 proxier.go:552] "Failed to open portal" err="can't open node port for <nil>:30615/TCP: listen tcp :30615: bind: address already in use" serviceName="metallb-system/nginx"
E1010 06:38:35.004706       1 proxier.go:552] "Failed to open portal" err="can't open node port for <nil>:30550/TCP: listen tcp :30550: bind: address already in use" serviceName="vault/vault:vault"
E1010 06:38:35.027619       1 proxier.go:552] "Failed to open portal" err="can't open node port for <nil>:30738/TCP: listen tcp :30738: bind: address already in use" serviceName="vault/vault:vault-cluster"
E1010 06:38:35.062077       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30738/TCP: listen tcp :30738: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1010 06:38:35.074164       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31163/TCP: listen tcp :31163: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1010 06:38:35.090126       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30615/TCP: listen tcp :30615: bind: address already in use" servicePortName="metallb-system/nginx"
E1010 06:38:35.113323       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30972/TCP: listen tcp :30972: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1010 06:38:35.141939       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30550/TCP: listen tcp :30550: bind: address already in use" servicePortName="vault/vault:vault"
E1010 06:38:35.214655       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30738/TCP: listen tcp :30738: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1010 06:38:35.227935       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31163/TCP: listen tcp :31163: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1010 06:38:35.245275       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30615/TCP: listen tcp :30615: bind: address already in use" servicePortName="metallb-system/nginx"
E1010 06:38:35.272512       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30972/TCP: listen tcp :30972: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1010 06:38:35.297487       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30550/TCP: listen tcp :30550: bind: address already in use" servicePortName="vault/vault:vault"
I1010 06:38:44.381790       1 tunnel.go:118] [MDNS] Discovery found peer: {12D3KooWNLAPNyViyXgHyAoTtvPc9D8fW3R5RJCAF8iiwpALCUQY: [/ip4/127.0.0.1/tcp/20006 /ip4/10.5.0.3/tcp/20006]}
I1010 06:38:44.381987       1 tunnel.go:130] [MDNS] New stream between peer {12D3KooWNLAPNyViyXgHyAoTtvPc9D8fW3R5RJCAF8iiwpALCUQY: [/ip4/127.0.0.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/169.254.96.16/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.5.0.3/tcp/20006]} success
I1010 06:38:44.383678       1 tunnel.go:166] [MDNS] Discovery to default-worker2 : {12D3KooWNLAPNyViyXgHyAoTtvPc9D8fW3R5RJCAF8iiwpALCUQY: [/ip4/127.0.0.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/169.254.96.16/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.5.0.3/tcp/20006]}
I1010 06:38:44.384062       1 tunnel.go:175] Discovery service got a new stream from {12D3KooWNLAPNyViyXgHyAoTtvPc9D8fW3R5RJCAF8iiwpALCUQY: [/ip4/10.5.0.3/tcp/20006]}
I1010 06:38:44.384175       1 tunnel.go:204] [MDNS] Discovery from default-worker2 : {12D3KooWNLAPNyViyXgHyAoTtvPc9D8fW3R5RJCAF8iiwpALCUQY: [/ip4/10.5.0.3/tcp/20006]}
E1010 06:39:04.470536       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31163/TCP: listen tcp :31163: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1010 06:39:04.509817       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30615/TCP: listen tcp :30615: bind: address already in use" servicePortName="metallb-system/nginx"
E1010 06:39:04.578256       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30738/TCP: listen tcp :30738: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1010 06:39:04.604133       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30972/TCP: listen tcp :30972: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1010 06:39:04.730338       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30550/TCP: listen tcp :30550: bind: address already in use" servicePortName="vault/vault:vault"
E1010 06:39:34.835684       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30550/TCP: listen tcp :30550: bind: address already in use" servicePortName="vault/vault:vault"
E1010 06:39:34.924507       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30738/TCP: listen tcp :30738: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1010 06:39:34.944426       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31163/TCP: listen tcp :31163: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1010 06:39:34.979393       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30615/TCP: listen tcp :30615: bind: address already in use" servicePortName="metallb-system/nginx"
E1010 06:39:35.024317       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30972/TCP: listen tcp :30972: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1010 06:40:05.104791       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30738/TCP: listen tcp :30738: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1010 06:40:05.116168       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31163/TCP: listen tcp :31163: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1010 06:40:05.134939       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30615/TCP: listen tcp :30615: bind: address already in use" servicePortName="metallb-system/nginx"
E1010 06:40:05.156607       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30972/TCP: listen tcp :30972: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1010 06:40:05.193246       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30550/TCP: listen tcp :30550: bind: address already in use" servicePortName="vault/vault:vault"
E1010 06:40:35.300506       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30738/TCP: listen tcp :30738: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1010 06:40:35.323012       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31163/TCP: listen tcp :31163: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1010 06:40:35.364481       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30615/TCP: listen tcp :30615: bind: address already in use" servicePortName="metallb-system/nginx"
E1010 06:40:35.415594       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30972/TCP: listen tcp :30972: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1010 06:40:35.434967       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30550/TCP: listen tcp :30550: bind: address already in use" servicePortName="vault/vault:vault"

Moreover: When i deploy a pod and try to resolve to any hostname, both external and cluster local service resolution is broken, when edgemesh is rolled out.

This is a try before rolling out edgemesh

$ kubectl run alpine --rm -ti --image=alpine -- /bin/sh
Found existing alias for "kubectl". You should use: "k"
If you don't see a command prompt, try pressing enter.
/ # nslookup www.google.com
Server:         10.96.0.10
Address:        10.96.0.10:53

Non-authoritative answer:
Name:   www.google.com
Address: 142.250.186.36

Non-authoritative answer:
Name:   www.google.com
Address: 2a00:1450:4001:827::2004

/ # nslookup cloudcore.kubeedge.svc.cluster.local
Server:         10.96.0.10
Address:        10.96.0.10:53


Name:   cloudcore.kubeedge.svc.cluster.local
Address: 10.96.163.34

This is the same try, when edgemesh is rolled out and gives the errors above:

$ kubectl run alpine --rm -ti --image=alpine -- /bin/sh
Found existing alias for "kubectl". You should use: "k"
If you don't see a command prompt, try pressing enter.
/ # nslookup www.google.com
;; connection timed out; no servers could be reached

/ # nslookup cloudcore.kubeedge.svc.cluster.local
;; connection timed out; no servers could be reached

What you expected to happen:

I'd expect that edgemesh does not show such errors. I'd expect that DNS resolution still works, even if edgemesh has some errors. This totaly breaks the clusterwide DNS resolution!

How to reproduce it (as minimally and precisely as possible): I guess, just run a kind cluster and deploy the helm chart(?)

Anything else we need to know?:

Environment:

EdgeMesh version: v1.12.0
Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.9", GitCommit:"6df4433e288edc9c40c2e344eb336f63fad45cd2", GitTreeState:"clean", BuildDate:"2022-05-19T19:53:08Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

KubeEdge version(e.g. cloudcore --version and edgecore --version): 1.11.0

Oct 10 '22 06:10 siredmar

Failed to ensure portal

This log does not affect EdgeMesh usage. This error log just because both kube-proxy and edgemesh handle nodePort-type services.

Oct 10 '22 07:10 Poorunga

Can you print some edgemesh-agent logs when you exec nslookup xxx command?

Oct 10 '22 07:10 Poorunga

Can you print some edgemesh-agent logs when you exec nslookup xxx command?

Sure,

E1010 08:41:42.400357       1 loadbalancer.go:683] "Dial failed" err="get proxy stream from default-control-plane error: new stream between default-control-plane err: routing: not found"
E1010 08:41:42.902339       1 loadbalancer.go:683] "Dial failed" err="get proxy stream from default-control-plane error: new stream between default-control-plane err: routing: not found"
E1010 08:41:43.904170       1 loadbalancer.go:683] "Dial failed" err="get proxy stream from default-control-plane error: new stream between default-control-plane err: routing: not found"
E1010 08:41:45.905801       1 loadbalancer.go:683] "Dial failed" err="get proxy stream from default-control-plane error: new stream between default-control-plane err: routing: not found"
E1010 08:41:46.157257       1 loadbalancer.go:683] "Dial failed" err="get proxy stream from default-control-plane error: new stream between default-control-plane err: routing: not found"

Oct 10 '22 08:10 siredmar

There are two ways to fix your problem, you can choose one:

deploy edgemesh-agent to your master node(default-control-plane):

kubectl taint nodes --all node-role.kubernetes.io/master-

Filter kube-dns service and don't let edgemesh proxy it:

kubectl -n kube-system label services kube-dns service.edgemesh.kubeedge.io/service-proxy-name=""

Oct 10 '22 08:10 Poorunga

@Poorunga thanks! I think the issue is solved by ignoring the kube-dns service. I opened a new issue #389 that targets the original problem with the setup.

Oct 10 '22 10:10 siredmar

@Poorunga I just provisioned a RPI device and see strange behavior when resolving to both service and external hosts

Here you see logs of nslookup on www.google.com from the issuing pod and edgecore running on that node

Alpine Pod:

/ $ nslookup www.google.de
Server:		169.254.96.16
Address:	169.254.96.16:53

;; connection timed out; no servers could be reached

edgemesh-agent:

I1011 09:35:44.288815       1 log.go:184] [INFO] 172.17.0.2:42763 - 24713 "AAAA IN www.google.de. udp 31 false 512" NOERROR - 0 0.005642376s
I1011 09:35:44.288936       1 log.go:184] [ERROR] plugin/errors: 2 www.google.de. AAAA: dial tcp 10.96.0.10:53: connect: network is unreachable

Resolving to echo-svc.local-service.svc.cluster.local and echo-svc.local-service.svc

Alpine pod:

/ $ nslookup echo-svc.local-service.svc.cluster.local
;; connection timed out; no servers could be reached

/ $ nslookup echo-svc.local-service.svc
Server:		169.254.96.16
Address:	169.254.96.16:53

;; connection timed out; no servers could be reached

edgemesh-agent:

I1011 09:38:54.041490       1 log.go:184] [INFO] 172.17.0.2:46341 - 62615 "A IN echo-svc.local-service.svc.cluster.local. udp 58 false 512" NOERROR - 0 1.03132987s
I1011 09:38:54.041621       1 log.go:184] [ERROR] plugin/errors: 2 echo-svc.local-service.svc.cluster.local. A: dial tcp 10.96.0.10:53: connect: network is unreachable
I1011 09:38:54.048231       1 log.go:184] [INFO] 172.17.0.2:46341 - 62615 "A IN echo-svc.local-service.svc.cluster.local. udp 58 false 512" NOERROR - 0 0.005393157s
I1011 09:38:54.048361       1 log.go:184] [ERROR] plugin/errors: 2 echo-svc.local-service.svc.cluster.local. A: dial tcp 10.96.0.10:53: connect: network is unreachable
I1011 09:38:55.516287       1 log.go:184] [INFO] 172.17.0.2:46341 - 64263 "AAAA IN echo-svc.local-service.svc.cluster.local. udp 58 false 512" NOERROR - 0 0.005235623s
I1011 09:38:55.516418       1 log.go:184] [ERROR] plugin/errors: 2 echo-svc.local-service.svc.cluster.local. AAAA: dial tcp 10.96.0.10:53: connect: network is unreachable
I1011 09:38:56.056650       1 log.go:184] [INFO] 172.17.0.2:46341 - 64263 "AAAA IN echo-svc.local-service.svc.cluster.local. udp 58 false 512" NOERROR - 0 3.04646718s
I1011 09:38:56.056799       1 log.go:184] [ERROR] plugin/errors: 2 echo-svc.local-service.svc.cluster.local. AAAA: dial tcp 10.96.0.10:53: connect: network is unreachable
I1011 09:38:57.081943       1 log.go:184] [INFO] 172.17.0.2:46341 - 62615 "A IN echo-svc.local-service.svc.cluster.local. udp 58 false 512" NOERROR - 0 3.032367564s
I1011 09:38:57.082094       1 log.go:184] [ERROR] plugin/errors: 2 echo-svc.local-service.svc.cluster.local. A: dial tcp 10.96.0.10:53: connect: network is unreachable


I1011 09:39:13.880523       1 log.go:184] [INFO] 172.17.0.2:44624 - 25263 "A IN echo-svc.local-service.svc. udp 44 false 512" NOERROR - 0 7.244601082s
I1011 09:39:13.880671       1 log.go:184] [ERROR] plugin/errors: 2 echo-svc.local-service.svc. A: dial tcp 10.96.0.10:53: connect: network is unreachable
I1011 09:39:14.730599       1 log.go:184] [INFO] 172.17.0.2:44624 - 26800 "AAAA IN echo-svc.local-service.svc. udp 44 false 512" NOERROR - 0 7.088760533s
I1011 09:39:14.730692       1 log.go:184] [ERROR] plugin/errors: 2 echo-svc.local-service.svc. AAAA: dial tcp 10.96.0.10:53: i/o timeout
I1011 09:39:15.004193       1 log.go:184] [INFO] 172.17.0.2:46341 - 62615 "A IN echo-svc.local-service.svc.cluster.local. udp 58 false 512" NOERROR - 0 19.493157444s
I1011 09:39:15.004271       1 log.go:184] [ERROR] plugin/errors: 2 echo-svc.local-service.svc.cluster.local. A: dial tcp 10.96.0.10:53: i/o timeout
I1011 09:39:16.156745       1 log.go:184] [INFO] 172.17.0.2:43523 - 44372 "AAAA IN echo-svc.local-service.svc. udp 44 false 512" NOERROR - 0 0.006346069s
I1011 09:39:16.156890       1 log.go:184] [ERROR] plugin/errors: 2 echo-svc.local-service.svc. AAAA: dial tcp 10.96.0.10:53: connect: network is unreachable
I1011 09:39:16.184136       1 log.go:184] [INFO] 172.17.0.2:44624 - 26800 "AAAA IN echo-svc.local-service.svc. udp 44 false 512" NOERROR - 0 7.053448929s
I1011 09:39:16.184275       1 log.go:184] [ERROR] plugin/errors: 2 echo-svc.local-service.svc. AAAA: dial tcp 10.96.0.10:53: connect: network is unreachable
I1011 09:39:16.696716       1 log.go:184] [INFO] 172.17.0.2:43523 - 42780 "A IN echo-svc.local-service.svc. udp 44 false 512" NOERROR - 0 3.04922104s
I1011 09:39:16.696805       1 log.go:184] [ERROR] plugin/errors: 2 echo-svc.local-service.svc. A: dial tcp 10.96.0.10:53: connect: network is unreachable
I1011 09:39:16.704284       1 log.go:184] [INFO] 172.17.0.2:43523 - 42780 "A IN echo-svc.local-service.svc. udp 44 false 512" NOERROR - 0 0.006847431s
I1011 09:39:16.704395       1 log.go:184] [ERROR] plugin/errors: 2 echo-svc.local-service.svc. A: dial tcp 10.96.0.10:53: connect: network is unreachable
I1011 09:39:20.692612       1 log.go:184] [INFO] 172.17.0.2:44624 - 25263 "A IN echo-svc.local-service.svc. udp 44 false 512" NOERROR - 0 11.561934099s
I1011 09:39:20.692698       1 log.go:184] [ERROR] plugin/errors: 2 echo-svc.local-service.svc. A: dial tcp 10.96.0.10:53: i/o timeout
I1011 09:39:23.356812       1 log.go:184] [INFO] 172.17.0.2:43523 - 42780 "A IN echo-svc.local-service.svc. udp 44 false 512" NOERROR - 0 7.206428595s
I1011 09:39:23.356932       1 log.go:184] [ERROR] plugin/errors: 2 echo-svc.local-service.svc. A: dial tcp 10.96.0.10:53: connect: network is unreachable
I1011 09:39:23.357074       1 log.go:184] [INFO] 172.17.0.2:43523 - 44372 "AAAA IN echo-svc.local-service.svc. udp 44 false 512" NOERROR - 0 7.197426623s
I1011 09:39:23.357177       1 log.go:184] [ERROR] plugin/errors: 2 echo-svc.local-service.svc. AAAA: dial tcp 10.96.0.10:53: connect: network is unreachable
I1011 09:39:23.863673       1 log.go:184] [INFO] 172.17.0.2:43523 - 42780 "A IN echo-svc.local-service.svc. udp 44 false 512" NOERROR - 0 7.15858128s
I1011 09:39:23.863765       1 log.go:184] [ERROR] plugin/errors: 2 echo-svc.local-service.svc. A: dial tcp 10.96.0.10:53: connect: network is unreachable

Oct 11 '22 09:10 siredmar

Are you enabled the cacheDNS?

Oct 13 '22 03:10 Poorunga

Yes, i enabled cacheDNS

Oct 13 '22 09:10 siredmar

Overall setup

I try to describe what my current setup looks like.

I have a kind running locally on my laptop. The IP Address is 192.168.1.42. I have a RPi device running in my local WIFI with the IP 192.168.1.100.

On my laptop i configured the iptables, that every access to 192.168.1.42 on several ports i use to forward to localhost. I then run kubectl port-forward on every of these ports. This way i can connect the edgecore running on my RPi to my cloudcore registering as node. This works flawlessly.

$ sudo iptables -t nat -L PREROUTING

[sudo] password for armin: 
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL
DNAT       tcp  --  anywhere             192.168.1.0/24       tcp dpt:webmin to:127.0.0.1:10000
DNAT       tcp  --  anywhere             192.168.1.0/24       tcp dpt:10002 to:127.0.0.1:10002
DNAT       tcp  --  anywhere             192.168.1.0/24       tcp dpt:10003 to:127.0.0.1:10003
DNAT       tcp  --  anywhere             192.168.1.0/24       tcp dpt:10004 to:127.0.0.1:10004
DNAT       tcp  --  anywhere             192.168.1.0/24       tcp dpt:20006 to:127.0.0.1:20006
DNAT       tcp  --  anywhere             192.168.1.0/24       tcp dpt:8200 to:127.0.0.1:8200

$ ps -ef | grep -v grep | grep "kubectl port-"
Found existing global alias for "| grep". You should use: "G"
armin    2857583       1  0 10:00 ?        00:00:03 kubectl port-forward svc/cloudcore -n kubeedge 10000:10000
armin    2857604       1  0 10:00 ?        00:00:02 kubectl port-forward svc/cloudcore -n kubeedge 10002:10002
armin    2857624       1  0 10:00 ?        00:00:02 kubectl port-forward svc/cloudcore -n kubeedge 10003:10003
armin    2857647       1  0 10:00 ?        00:00:04 kubectl port-forward svc/cloudcore -n kubeedge 10004:10004
armin    2857669       1  0 10:00 ?        00:00:02 kubectl port-forward svc/edgemesh-cloud-agent -n kubeedge 20006:20006
armin    2857693       1  0 10:00 ?        00:00:02 kubectl port-forward svc/vault -n vault 8200:8200

My kind cluster does have one control-plane and two worker nodes. Here you see a list of nodes with my RPi device (called clownfish). The kind cluster uses a docker network that is configured to use 10.5.0.x as node IPs.

k get nodes -o wide          
NAME                    STATUS   ROLES                  AGE   VERSION                                            INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
clownfish               Ready    agent,edge             20h   v1.22.6-kubeedge-v1.11.1-12+0d66acb85546eb-dirty   192.168.1.100   <none>        Ubuntu 22.04.1 LTS   5.15.0-1012-raspi   docker://20.10.18
default-control-plane   Ready    control-plane,master   20h   v1.22.9                                            10.5.0.4        <none>        Ubuntu 21.10         5.17.0-1016-oem     containerd://1.6.4
default-worker          Ready    <none>                 20h   v1.22.9                                            10.5.0.2        <none>        Ubuntu 21.10         5.17.0-1016-oem     containerd://1.6.4
default-worker2         Ready    <none>                 20h   v1.22.9                                            10.5.0.3        <none>        Ubuntu 21.10         5.17.0-1016-oem     containerd://1.6.4

What i tried

So, i've played around a bit with the edgemesh helm chart and deployed it with several configurations. In no case the edgemesh-agent running on my RPi can connect to any Relay. I even split up two helm deployments. One for edge, one for cloud. The cloud deployment has a service, that i can iptable and port-forward to. Then my RPi connects to 192.168.1.42:20006 as a relay.

I filtered out two services

kubectl label services -n default kubernetes service.edgemesh.kubeedge.io/service-proxy-name="" 
kubectl label services -n kube-system kube-dns service.edgemesh.kubeedge.io/service-proxy-name=""

My configuration looks like this. In my current setup they both look identical for edge and cloud deployment. However, the cloud deployment has the service i use my iptables&port-forward mechanism on.

    modules:
      edgeDNS:
        enable: true
        cacheDNS:
          enable: true
      edgeProxy:
        enable: true
      edgeTunnel:
        enable: true
        relayNodes:
        - nodeName: default-worker
          advertiseAddress:
          - 10.5.0.2
          - 192.168.1.42

These are the pods running in my kubeedge namespace.

NAME                         READY   STATUS    RESTARTS   AGE     IP              NODE                    NOMINATED NODE   READINESS GATES
cloudcore-d5c5f4c45-fmx7p    2/2     Running   0          133m    10.5.0.3        default-worker2         <none>           <none>
edgemesh-cloud-agent-2q49f   1/1     Running   0          15m     10.5.0.3        default-worker2         <none>           <none>
edgemesh-cloud-agent-zf5j7   1/1     Running   0          15m     10.5.0.2        default-worker          <none>           <none>
edgemesh-edge-agent-knf4l    1/1     Running   0          15m     192.168.1.100   clownfish               <none>           <none>
iptables-manager-z8frg       1/1     Running   0          20h     10.5.0.4        default-control-plane   <none>           <none>

Using this setup i can verify that an alpine pod running on any cloud node can resolve to everything, both local services and external addresses. However, on the edge device i get the following errors from edgemesh-agent

k logs edgemesh-edge-agent-knf4l    
I1013 09:58:34.585175       1 server.go:55] Version: v1.12.0-dirty
I1013 09:58:34.585437       1 server.go:89] [1] Prepare agent to run
I1013 09:58:34.594571       1 netif.go:96] bridge device edgemesh0 already exists
I1013 09:58:34.595127       1 server.go:93] edgemesh-agent running on EdgeMode
I1013 09:58:34.595220       1 server.go:96] [2] New clients
I1013 09:58:34.656133       1 server.go:103] [3] Register beehive modules
I1013 09:58:34.905664       1 corefile.go:200] Automatically detect cluster dns: [10.96.0.10]
I1013 09:58:34.905774       1 corefile.go:151] nodelocal dns upstream servers: [10.96.0.10]
I1013 09:58:34.906639       1 module.go:34] Module EdgeDNS registered successfully
I1013 09:58:34.955710       1 server.go:66] Using userspace Proxier.
I1013 09:58:35.361859       1 module.go:34] Module EdgeProxy registered successfully
I1013 09:58:35.458982       1 module.go:159] I'm {12D3KooWRFoxFy8Esso4ymzwXdojgdwYGtCrKh3JYEppeR8mmQAj: [/ip4/192.168.1.100/tcp/20006 /ip4/127.0.0.1/tcp/20006]}
I1013 09:58:35.461051       1 module.go:181] Bootstrapping the DHT
I1013 09:58:35.461452       1 tunnel.go:387] [Bootstrap] bootstrapping to 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw
E1013 09:58:38.429775       1 tunnel.go:391] [Bootstrap] failed to bootstrap with {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/192.168.1.42/tcp/20006 /ip4/10.5.0.2/tcp/20006]}: failed to dial 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw:
  * [/ip4/192.168.1.42/tcp/20006] failed to negotiate security protocol: privnet: could not read full nonce
  * [/ip4/10.5.0.2/tcp/20006] dial tcp4 10.5.0.2:20006: connect: network is unreachable
E1013 09:58:38.430359       1 tunnel.go:402] [Bootstrap] Not all bootstrapDail connected, continue bootstrapDail...
I1013 09:58:48.430727       1 tunnel.go:387] [Bootstrap] bootstrapping to 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw
E1013 09:58:53.433077       1 tunnel.go:391] [Bootstrap] failed to bootstrap with {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/192.168.1.42/tcp/20006 /ip4/10.5.0.2/tcp/20006]}: failed to dial 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw:
  * [/ip4/192.168.1.42/tcp/20006] failed to negotiate security protocol: privnet: could not read full nonce
  * [/ip4/10.5.0.2/tcp/20006] dial tcp4 0.0.0.0:20006->10.5.0.2:20006: i/o timeout
E1013 09:58:53.433427       1 tunnel.go:402] [Bootstrap] Not all bootstrapDail connected, continue bootstrapDail...
I1013 09:58:58.431085       1 tunnel.go:387] [Bootstrap] bootstrapping to 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw
E1013 09:59:03.433644       1 tunnel.go:391] [Bootstrap] failed to bootstrap with {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/192.168.1.42/tcp/20006 /ip4/10.5.0.2/tcp/20006]}: failed to dial 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw:
  * [/ip4/10.5.0.2/tcp/20006] dial backoff
  * [/ip4/192.168.1.42/tcp/20006] failed to negotiate security protocol: context deadline exceeded
E1013 09:59:03.434092       1 tunnel.go:402] [Bootstrap] Not all bootstrapDail connected, continue bootstrapDail...
I1013 09:59:08.431474       1 tunnel.go:387] [Bootstrap] bootstrapping to 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw
E1013 09:59:13.433022       1 tunnel.go:391] [Bootstrap] failed to bootstrap with {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/192.168.1.42/tcp/20006 /ip4/10.5.0.2/tcp/20006]}: failed to dial 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw:
  * [/ip4/192.168.1.42/tcp/20006] dial backoff
  * [/ip4/10.5.0.2/tcp/20006] dial tcp4 0.0.0.0:20006->10.5.0.2:20006: i/o timeout
E1013 09:59:13.433280       1 tunnel.go:402] [Bootstrap] Not all bootstrapDail connected, continue bootstrapDail...
I1013 09:59:18.431336       1 tunnel.go:387] [Bootstrap] bootstrapping to 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw
E1013 09:59:18.441033       1 tunnel.go:391] [Bootstrap] failed to bootstrap with {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/192.168.1.42/tcp/20006 /ip4/10.5.0.2/tcp/20006]}: failed to dial 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw:
  * [/ip4/10.5.0.2/tcp/20006] dial backoff
  * [/ip4/192.168.1.42/tcp/20006] failed to negotiate security protocol: privnet: could not read full nonce
E1013 09:59:18.441428       1 tunnel.go:402] [Bootstrap] Not all bootstrapDail connected, continue bootstrapDail...
I1013 09:59:28.431189       1 tunnel.go:387] [Bootstrap] bootstrapping to 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw
E1013 09:59:33.433374       1 tunnel.go:391] [Bootstrap] failed to bootstrap with {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/192.168.1.42/tcp/20006 /ip4/10.5.0.2/tcp/20006]}: failed to dial 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw:
  * [/ip4/192.168.1.42/tcp/20006] dial backoff
  * [/ip4/10.5.0.2/tcp/20006] dial tcp4 0.0.0.0:20006->10.5.0.2:20006: i/o timeout
E1013 09:59:33.433756       1 tunnel.go:402] [Bootstrap] Not all bootstrapDail connected, continue bootstrapDail...
I1013 09:59:38.431122       1 tunnel.go:387] [Bootstrap] bootstrapping to 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw
E1013 09:59:43.435077       1 tunnel.go:391] [Bootstrap] failed to bootstrap with {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/192.168.1.42/tcp/20006 /ip4/10.5.0.2/tcp/20006]}: failed to dial 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw:
  * [/ip4/10.5.0.2/tcp/20006] dial backoff
  * [/ip4/192.168.1.42/tcp/20006] failed to negotiate security protocol: context deadline exceeded
E1013 09:59:43.435415       1 tunnel.go:402] [Bootstrap] Not all bootstrapDail connected, continue bootstrapDail...
I1013 09:59:43.435661       1 tunnel.go:387] [Bootstrap] bootstrapping to 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw
E1013 09:59:43.436311       1 tunnel.go:391] [Bootstrap] failed to bootstrap with {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/192.168.1.42/tcp/20006 /ip4/10.5.0.2/tcp/20006]}: failed to dial 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw:
  * [/ip4/192.168.1.42/tcp/20006] dial backoff
  * [/ip4/10.5.0.2/tcp/20006] dial backoff
E1013 09:59:43.436587       1 tunnel.go:402] [Bootstrap] Not all bootstrapDail connected, continue bootstrapDail...
W1013 09:59:43.436648       1 tunnel.go:410] [Bootstrap] bootstrapping to default-worker : {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/192.168.1.42/tcp/20006 /ip4/10.5.0.2/tcp/20006]} timeout, delete it from relayMap
W1013 09:59:43.436792       1 module.go:192] Failed to connect bootstrap: timed out waiting for the condition
I1013 09:59:43.439160       1 tunnel.go:63] Starting MDNS discovery service
I1013 09:59:43.439265       1 tunnel.go:76] Starting DHT discovery service
I1013 09:59:43.439470       1 module.go:34] Module EdgeTunnel registered successfully
I1013 09:59:43.439595       1 server.go:109] [4] Start all modules
I1013 09:59:43.440007       1 core.go:24] Starting module EdgeDNS
I1013 09:59:43.446721       1 core.go:24] Starting module EdgeProxy
I1013 09:59:43.447643       1 tunnel.go:447] Starting relay finder
I1013 09:59:43.448321       1 dns.go:32] Runs CoreDNS v1.8.0 as a cache dns
I1013 09:59:43.449607       1 config.go:317] "Starting service config controller"
I1013 09:59:43.449722       1 shared_informer.go:240] Waiting for caches to sync for service config
I1013 09:59:43.450326       1 config.go:135] "Starting endpoints config controller"
I1013 09:59:43.450391       1 shared_informer.go:240] Waiting for caches to sync for endpoints config
.:53 on 169.254.96.16
I1013 09:59:43.454700       1 log.go:184] [INFO] plugin/reload: Running configuration MD5 = f221339a42605d46dfa917055e01cc69
I1013 09:59:43.459760       1 loadbalancer.go:239] "Starting loadBalancer destinationRule controller"
I1013 09:59:43.459841       1 shared_informer.go:240] Waiting for caches to sync for loadBalancer destinationRule
I1013 09:59:43.462360       1 core.go:24] Starting module EdgeTunnel
I1013 09:59:43.574078       1 shared_informer.go:247] Caches are synced for loadBalancer destinationRule 
I1013 09:59:43.656269       1 shared_informer.go:247] Caches are synced for endpoints config 
I1013 09:59:43.656607       1 shared_informer.go:247] Caches are synced for service config 
I1013 09:59:43.970246       1 proxier.go:895] "Opened iptables from-containers public port for service" servicePortName="metallb-system/nginx" protocol=TCP nodePort=30694
I1013 09:59:43.984045       1 proxier.go:906] "Opened iptables from-host public port for service" servicePortName="metallb-system/nginx" protocol=TCP nodePort=30694
I1013 09:59:44.000151       1 proxier.go:916] "Opened iptables from-non-local public port for service" servicePortName="metallb-system/nginx" protocol=TCP nodePort=30694
I1013 09:59:44.331660       1 proxier.go:895] "Opened iptables from-containers public port for service" servicePortName="vault/vault:vault" protocol=TCP nodePort=31360
I1013 09:59:44.360560       1 proxier.go:906] "Opened iptables from-host public port for service" servicePortName="vault/vault:vault" protocol=TCP nodePort=31360
I1013 09:59:44.380364       1 proxier.go:916] "Opened iptables from-non-local public port for service" servicePortName="vault/vault:vault" protocol=TCP nodePort=31360
I1013 09:59:44.475265       1 proxier.go:895] "Opened iptables from-containers public port for service" servicePortName="vault/vault:vault-cluster" protocol=TCP nodePort=30976
I1013 09:59:44.492131       1 proxier.go:906] "Opened iptables from-host public port for service" servicePortName="vault/vault:vault-cluster" protocol=TCP nodePort=30976
I1013 09:59:44.510316       1 proxier.go:916] "Opened iptables from-non-local public port for service" servicePortName="vault/vault:vault-cluster" protocol=TCP nodePort=30976
I1013 09:59:44.681448       1 proxier.go:895] "Opened iptables from-containers public port for service" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" protocol=TCP nodePort=32227
I1013 09:59:44.697579       1 proxier.go:906] "Opened iptables from-host public port for service" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" protocol=TCP nodePort=32227
I1013 09:59:44.720520       1 proxier.go:916] "Opened iptables from-non-local public port for service" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" protocol=TCP nodePort=32227
I1013 09:59:44.813345       1 proxier.go:895] "Opened iptables from-containers public port for service" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" protocol=TCP nodePort=30715
I1013 09:59:44.829117       1 proxier.go:906] "Opened iptables from-host public port for service" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" protocol=TCP nodePort=30715
I1013 09:59:44.847629       1 proxier.go:916] "Opened iptables from-non-local public port for service" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" protocol=TCP nodePort=30715
I1013 10:00:02.904504       1 log.go:184] [INFO] 169.254.96.16:37889 - 49262 "HINFO IN 3333630752541257207.5062809213743632370. udp 57 false 512" NOERROR - 0 16.446867278s
I1013 10:00:02.904648       1 log.go:184] [ERROR] plugin/errors: 2 3333630752541257207.5062809213743632370. HINFO: dial tcp 10.96.0.10:53: connect: network is unreachable
I1013 10:00:04.952632       1 log.go:184] [INFO] 169.254.96.16:51182 - 44431 "HINFO IN 3333630752541257207.5062809213743632370. udp 57 false 512" NOERROR - 0 15.491938197s
I1013 10:00:04.952745       1 log.go:184] [ERROR] plugin/errors: 2 3333630752541257207.5062809213743632370. HINFO: dial tcp 10.96.0.10:53: connect: network is unreachable
I1013 10:00:05.496184       1 log.go:184] [INFO] 169.254.96.16:45436 - 58218 "HINFO IN 3333630752541257207.5062809213743632370. udp 57 false 512" NOERROR - 0 1.025121569s
I1013 10:00:05.496340       1 log.go:184] [ERROR] plugin/errors: 2 3333630752541257207.5062809213743632370. HINFO: dial tcp 10.96.0.10:53: connect: network is unreachable
I1013 10:00:13.457470       1 log.go:184] [INFO] 169.254.96.16:60045 - 58935 "HINFO IN 3333630752541257207.5062809213743632370. udp 57 false 512" NOERROR - 0 30.000642073s
I1013 10:00:13.457603       1 log.go:184] [ERROR] plugin/errors: 2 3333630752541257207.5062809213743632370. HINFO: dial tcp 10.96.0.10:53: i/o timeout
I1013 10:00:15.474005       1 log.go:184] [INFO] 169.254.96.16:47880 - 54869 "HINFO IN 3333630752541257207.5062809213743632370. udp 57 false 512" NOERROR - 0 23.011854279s
I1013 10:00:15.474099       1 log.go:184] [ERROR] plugin/errors: 2 3333630752541257207.5062809213743632370. HINFO: dial tcp 10.96.0.10:53: i/o timeout
I1013 10:00:18.475473       1 log.go:184] [INFO] 169.254.96.16:59401 - 35344 "HINFO IN 3333630752541257207.5062809213743632370. udp 57 false 512" NOERROR - 0 23.011897691s
I1013 10:00:18.475711       1 log.go:184] [ERROR] plugin/errors: 2 3333630752541257207.5062809213743632370. HINFO: dial tcp 10.96.0.10:53: i/o timeout
I1013 10:00:21.478334       1 log.go:184] [INFO] 169.254.96.16:58113 - 34864 "HINFO IN 3333630752541257207.5062809213743632370. udp 57 false 512" NOERROR - 0 23.012338837s
I1013 10:00:21.478479       1 log.go:184] [ERROR] plugin/errors: 2 3333630752541257207.5062809213743632370. HINFO: dial tcp 10.96.0.10:53: i/o timeout
I1013 10:00:24.480296       1 log.go:184] [INFO] 169.254.96.16:60392 - 36243 "HINFO IN 3333630752541257207.5062809213743632370. udp 57 false 512" NOERROR - 0 23.011644946s
I1013 10:00:24.480436       1 log.go:184] [ERROR] plugin/errors: 2 3333630752541257207.5062809213743632370. HINFO: dial tcp 10.96.0.10:53: i/o timeout

I see the error

E1013 09:59:43.435077       1 tunnel.go:391] [Bootstrap] failed to bootstrap with {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/192.168.1.42/tcp/20006 /ip4/10.5.0.2/tcp/20006]}: failed to dial 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw:
  * [/ip4/10.5.0.2/tcp/20006] dial backoff
  * [/ip4/192.168.1.42/tcp/20006] failed to negotiate security protocol: context deadline exceeded

I guess this is a libp2p related error message. I don't know how to come by with this error. The worst thing is, that when using the clusterDNS server (configured in edgecore.yaml) to be the edgemesh0 device, no pod can lookup any address. Neither svc addresses nor external addresses work.

When i try to resolv to vault.vault.svc.cluster.local in a pod running on my clownfish node.

I1013 10:23:16.806710       1 log.go:184] [INFO] 172.17.0.2:58791 - 16368 "A IN vault.vault.svc.cluster.local. udp 47 false 512" NOERROR - 0 0.005442524s
I1013 10:23:16.806792       1 log.go:184] [ERROR] plugin/errors: 2 vault.vault.svc.cluster.local. A: dial tcp 10.96.0.10:53: connect: network is unreachable
I1013 10:23:20.312152       1 log.go:184] [INFO] 172.17.0.2:58791 - 16368 "A IN vault.vault.svc.cluster.local. udp 47 false 512" NOERROR - 0 1.007592939s
I1013 10:23:20.312309       1 log.go:184] [ERROR] plugin/errors: 2 vault.vault.svc.cluster.local. A: dial tcp 10.96.0.10:53: connect: network is unreachable

When i try to resolv to www.google.com in a pod running on my clownfish node.

I1013 10:24:37.431713       1 log.go:184] [INFO] 172.17.0.2:59511 - 65421 "A IN www.google.com. udp 32 false 512" NOERROR - 0 1.021613818s
I1013 10:24:37.431833       1 log.go:184] [ERROR] plugin/errors: 2 www.google.com. A: dial tcp 10.96.0.10:53: connect: network is unreachable
I1013 10:24:37.438355       1 log.go:184] [INFO] 172.17.0.2:59511 - 65421 "A IN www.google.com. udp 32 false 512" NOERROR - 0 0.005274695s
I1013 10:24:37.438471       1 log.go:184] [ERROR] plugin/errors: 2 www.google.com. A: dial tcp 10.96.0.10:53: connect: network is unreachable

Other logs

Just to be complete here. These are the logs of both edgemesh-agents running on the cloud nodes.

default-worker

I1013 09:58:32.053014       1 server.go:55] Version: v1.12.0-dirty
I1013 09:58:32.053050       1 server.go:89] [1] Prepare agent to run
I1013 09:58:32.053465       1 netif.go:96] bridge device edgemesh0 already exists
I1013 09:58:32.053510       1 server.go:93] edgemesh-agent running on CloudMode
I1013 09:58:32.053517       1 server.go:96] [2] New clients
W1013 09:58:32.053524       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1013 09:58:32.054655       1 server.go:103] [3] Register beehive modules
I1013 09:58:32.064521       1 corefile.go:200] Automatically detect cluster dns: [10.96.0.10]
I1013 09:58:32.064539       1 corefile.go:151] nodelocal dns upstream servers: [10.96.0.10]
I1013 09:58:32.064737       1 module.go:34] Module EdgeDNS registered successfully
I1013 09:58:32.064984       1 server.go:66] Using userspace Proxier.
I1013 09:58:32.305397       1 module.go:34] Module EdgeProxy registered successfully
I1013 09:58:32.309347       1 module.go:159] I'm {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/10.5.0.2/tcp/20006 /ip4/127.0.0.1/tcp/20006 /ip4/10.5.0.2/tcp/20006 /ip4/192.168.1.42/tcp/20006]}
I1013 09:58:32.309388       1 module.go:168] Run as a relay node
I1013 09:58:32.309453       1 module.go:181] Bootstrapping the DHT
I1013 09:58:32.310044       1 tunnel.go:63] Starting MDNS discovery service
I1013 09:58:32.310061       1 tunnel.go:76] Starting DHT discovery service
I1013 09:58:32.310096       1 module.go:34] Module EdgeTunnel registered successfully
I1013 09:58:32.310106       1 server.go:109] [4] Start all modules
I1013 09:58:32.310158       1 core.go:24] Starting module EdgeDNS
I1013 09:58:32.310195       1 core.go:24] Starting module EdgeProxy
I1013 09:58:32.310219       1 core.go:24] Starting module EdgeTunnel
I1013 09:58:32.311695       1 tunnel.go:447] Starting relay finder
I1013 09:58:32.311730       1 dns.go:32] Runs CoreDNS v1.8.0 as a cache dns
.:53 on 169.254.96.16
I1013 09:58:32.312194       1 log.go:184] [INFO] plugin/reload: Running configuration MD5 = f221339a42605d46dfa917055e01cc69
I1013 09:58:32.312707       1 config.go:317] "Starting service config controller"
I1013 09:58:32.312721       1 shared_informer.go:240] Waiting for caches to sync for service config
I1013 09:58:32.312738       1 config.go:135] "Starting endpoints config controller"
I1013 09:58:32.312744       1 shared_informer.go:240] Waiting for caches to sync for endpoints config
I1013 09:58:32.313200       1 loadbalancer.go:239] "Starting loadBalancer destinationRule controller"
I1013 09:58:32.313214       1 shared_informer.go:240] Waiting for caches to sync for loadBalancer destinationRule
I1013 09:58:32.385583       1 log.go:184] [INFO] 169.254.96.16:47799 - 8673 "HINFO IN 3835683483591670212.5865939923289163945. udp 57 false 512" NXDOMAIN qr,rd,ra 132 0.072989245s
I1013 09:58:32.413110       1 shared_informer.go:247] Caches are synced for endpoints config 
I1013 09:58:32.413134       1 shared_informer.go:247] Caches are synced for service config 
I1013 09:58:32.413363       1 shared_informer.go:247] Caches are synced for loadBalancer destinationRule 
E1013 09:58:32.591324       1 proxier.go:552] "Failed to open portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" serviceName="metallb-system/nginx"
E1013 09:58:32.644012       1 proxier.go:552] "Failed to open portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" serviceName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 09:58:32.668116       1 proxier.go:552] "Failed to open portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" serviceName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 09:58:32.872456       1 proxier.go:552] "Failed to open portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" serviceName="vault/vault:vault"
E1013 09:58:32.911118       1 proxier.go:552] "Failed to open portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" serviceName="vault/vault:vault-cluster"
E1013 09:58:32.947920       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 09:58:33.006233       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 09:58:33.022859       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 09:58:33.048127       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 09:58:33.060834       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
I1013 09:58:47.214010       1 tunnel.go:118] [MDNS] Discovery found peer: {12D3KooWNLAPNyViyXgHyAoTtvPc9D8fW3R5RJCAF8iiwpALCUQY: [/ip4/10.5.0.3/tcp/20006 /ip4/127.0.0.1/tcp/20006]}
E1013 09:58:47.214613       1 tunnel.go:121] [MDNS] New stream between peer {12D3KooWNLAPNyViyXgHyAoTtvPc9D8fW3R5RJCAF8iiwpALCUQY: [/ip4/127.0.0.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/169.254.96.16/tcp/20006 /ip4/10.244.1.1/tcp/20006 /ip4/10.5.0.3/tcp/20006]} err: protocol not supported
I1013 09:58:47.227441       1 tunnel.go:175] Discovery service got a new stream from {12D3KooWNLAPNyViyXgHyAoTtvPc9D8fW3R5RJCAF8iiwpALCUQY: [/ip4/10.5.0.3/tcp/20006]}
I1013 09:58:47.227616       1 tunnel.go:204] [MDNS] Discovery from default-worker2 : {12D3KooWNLAPNyViyXgHyAoTtvPc9D8fW3R5RJCAF8iiwpALCUQY: [/ip4/10.5.0.3/tcp/20006]}
I1013 09:58:47.227996       1 tunnel.go:175] Discovery service got a new stream from {12D3KooWNLAPNyViyXgHyAoTtvPc9D8fW3R5RJCAF8iiwpALCUQY: [/ip4/10.5.0.3/tcp/20006]}
I1013 09:58:47.228050       1 tunnel.go:204] [DHT] Discovery from default-worker2 : {12D3KooWNLAPNyViyXgHyAoTtvPc9D8fW3R5RJCAF8iiwpALCUQY: [/ip4/10.5.0.3/tcp/20006]}
E1013 09:59:02.365724       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 09:59:02.392076       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 09:59:02.404663       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 09:59:02.416663       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 09:59:02.469876       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 09:59:32.614013       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 09:59:32.728518       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 09:59:32.788302       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 09:59:32.811715       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 09:59:32.833275       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:00:02.923837       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:00:03.016977       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:00:03.063378       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:00:03.117770       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:00:03.137553       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
I1013 10:00:04.813604       1 tunnel.go:289] Proxy service got a new stream from {12D3KooWNLAPNyViyXgHyAoTtvPc9D8fW3R5RJCAF8iiwpALCUQY: [/ip4/10.5.0.3/tcp/20006]}
I1013 10:00:04.814067       1 tunnel.go:339] Success proxy for {tcp default-worker 10.244.2.8:8200}
E1013 10:00:04.825960       1 conn.go:53] "src close failed" err="close tcp 10.244.2.1:50914->10.244.2.8:8200: use of closed network connection"
I1013 10:00:04.827728       1 tunnel.go:289] Proxy service got a new stream from {12D3KooWNLAPNyViyXgHyAoTtvPc9D8fW3R5RJCAF8iiwpALCUQY: [/ip4/10.5.0.3/tcp/20006]}
I1013 10:00:04.827940       1 tunnel.go:339] Success proxy for {tcp default-worker 10.244.2.8:8200}
E1013 10:00:33.221545       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:00:33.254524       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:00:33.292859       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:00:33.303790       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:00:33.314174       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:01:03.432850       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:01:03.539118       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:01:03.611634       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:01:03.655240       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:01:03.671249       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:01:33.716296       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:01:33.772180       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:01:33.812693       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:01:33.829830       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:01:33.855432       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:02:03.918242       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:02:03.977209       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:02:04.008615       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:02:04.023226       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:02:04.034360       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:02:34.120760       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:02:34.141925       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:02:34.157882       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:02:34.198164       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:02:34.260545       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:03:04.356770       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:03:04.454120       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:03:04.516131       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:03:04.543193       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:03:04.569471       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:03:34.633695       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:03:34.663362       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:03:34.689013       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:03:34.702576       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:03:34.722653       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:04:04.888038       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:04:04.932902       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:04:04.989423       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:04:05.002342       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:04:05.019857       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:04:35.131989       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:04:35.181031       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:04:35.206665       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:04:35.224440       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:04:35.296230       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:05:05.106776       1 conn.go:50] "dest close failed" err="close tcp 10.244.2.1:50930->10.244.2.8:8200: use of closed network connection"
E1013 10:05:05.393508       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:05:05.463198       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:05:05.527153       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:05:05.568336       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:05:05.598887       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:05:35.639042       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:05:35.651075       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:05:35.674051       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:05:35.684941       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:05:35.732086       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:06:05.816610       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:06:05.866735       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:06:05.905394       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:06:05.942025       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:06:05.953120       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:06:36.047123       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:06:36.135460       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:06:36.178213       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:06:36.210666       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:06:36.236946       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:07:06.326886       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:07:06.390916       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:07:06.415390       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:07:06.425891       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:07:06.445532       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:07:36.535145       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:07:36.601057       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:07:36.646800       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:07:36.659026       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:07:36.675057       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:08:06.840101       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:08:06.894940       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:08:06.913188       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:08:06.937418       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:08:06.997936       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:08:37.085236       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:08:37.115642       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:08:37.139223       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:08:37.156839       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:08:37.207644       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:09:07.314687       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:09:07.406789       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:09:07.457116       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:09:07.496006       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:09:07.509242       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:09:37.553544       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:09:37.616070       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:09:37.650007       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:09:37.676037       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:09:37.695003       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:10:07.776450       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:10:07.804928       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:10:07.827619       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:10:07.871491       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:10:07.943805       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:10:38.027819       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:10:38.093120       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:10:38.123286       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:10:38.144483       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:10:38.157268       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:11:08.233801       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:11:08.255774       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:11:08.268057       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:11:08.292595       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:11:08.338878       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:11:38.446243       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:11:38.561697       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:11:38.627823       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:11:38.689204       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:11:38.713219       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:12:08.794383       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:12:08.850341       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:12:08.888982       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:12:08.904335       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:12:08.923734       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:12:39.058807       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:12:39.121176       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:12:39.149917       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:12:39.174065       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:12:39.194023       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:13:09.255430       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:13:09.341316       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:13:09.386132       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:13:09.434831       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:13:09.459790       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:13:39.515826       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:13:39.554017       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:13:39.572189       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:13:39.594626       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:13:39.642538       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:14:09.746389       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:14:09.804980       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:14:09.829570       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:14:09.855302       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:14:09.929516       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:14:40.008907       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:14:40.034170       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:14:40.047172       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:14:40.058510       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:14:40.109227       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:15:10.193531       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:15:10.238292       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:15:10.263677       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:15:10.276137       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:15:10.293109       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:15:40.401930       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:15:40.452018       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:15:40.471171       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:15:40.514545       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:15:40.524783       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:16:10.652317       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:16:10.700653       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:16:10.752083       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:16:10.778508       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:16:10.807527       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:16:40.978976       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:16:40.993276       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"
E1013 10:16:41.005268       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:16:41.044249       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:16:41.102362       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:17:11.172157       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https"
E1013 10:17:11.230988       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http"
E1013 10:17:11.262200       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx"
E1013 10:17:11.293516       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault"
E1013 10:17:11.305629       1 proxier.go:422] "Failed to ensure portal" err="can't open node port for <nil>:30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster"

default-worker2

`` I1013 09:58:32.076358 1 server.go:55] Version: v1.12.0-dirty I1013 09:58:32.076393 1 server.go:89] [1] Prepare agent to run I1013 09:58:32.076519 1 netif.go:96] bridge device edgemesh0 already exists I1013 09:58:32.076558 1 server.go:93] edgemesh-agent running on CloudMode I1013 09:58:32.076564 1 server.go:96] [2] New clients W1013 09:58:32.076572 1 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I1013 09:58:32.076943 1 server.go:103] [3] Register beehive modules I1013 09:58:32.088886 1 corefile.go:200] Automatically detect cluster dns: [10.96.0.10] I1013 09:58:32.088911 1 corefile.go:151] nodelocal dns upstream servers: [10.96.0.10] I1013 09:58:32.089134 1 module.go:34] Module EdgeDNS registered successfully I1013 09:58:32.089427 1 server.go:66] Using userspace Proxier. I1013 09:58:32.201966 1 module.go:34] Module EdgeProxy registered successfully I1013 09:58:32.212241 1 module.go:159] I'm {12D3KooWNLAPNyViyXgHyAoTtvPc9D8fW3R5RJCAF8iiwpALCUQY: [/ip4/10.5.0.3/tcp/20006 /ip4/127.0.0.1/tcp/20006]} I1013 09:58:32.212462 1 module.go:181] Bootstrapping the DHT I1013 09:58:32.212553 1 tunnel.go:387] [Bootstrap] bootstrapping to 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw E1013 09:58:37.211552 1 tunnel.go:391] [Bootstrap] failed to bootstrap with {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/10.5.0.2/tcp/20006 /ip4/192.168.1.42/tcp/20006]}: failed to dial 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw:

[/ip4/10.5.0.2/tcp/20006] dial tcp4 10.5.0.2:20006: connect: connection refused
[/ip4/192.168.1.42/tcp/20006] dial tcp4 0.0.0.0:20006->192.168.1.42:20006: i/o timeout E1013 09:58:37.211693 1 tunnel.go:402] [Bootstrap] Not all bootstrapDail connected, continue bootstrapDail... I1013 09:58:47.212189 1 tunnel.go:387] [Bootstrap] bootstrapping to 12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw I1013 09:58:47.212226 1 tunnel.go:397] [Bootstrap] success bootstrapped with {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/10.5.0.2/tcp/20006 /ip4/192.168.1.42/tcp/20006]} I1013 09:58:47.212807 1 tunnel.go:63] Starting MDNS discovery service I1013 09:58:47.223490 1 tunnel.go:76] Starting DHT discovery service I1013 09:58:47.223571 1 module.go:34] Module EdgeTunnel registered successfully I1013 09:58:47.223586 1 server.go:109] [4] Start all modules I1013 09:58:47.223628 1 tunnel.go:447] Starting relay finder I1013 09:58:47.223771 1 dns.go:32] Runs CoreDNS v1.8.0 as a cache dns I1013 09:58:47.223996 1 core.go:24] Starting module EdgeDNS I1013 09:58:47.224074 1 core.go:24] Starting module EdgeProxy I1013 09:58:47.224130 1 core.go:24] Starting module EdgeTunnel .:53 on 169.254.96.16 I1013 09:58:47.224310 1 log.go:184] [INFO] plugin/reload: Running configuration MD5 = f221339a42605d46dfa917055e01cc69 I1013 09:58:47.224247 1 tunnel.go:118] [MDNS] Discovery found peer: {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/10.5.0.2/tcp/20006 /ip4/192.168.1.42/tcp/20006 /ip4/127.0.0.1/tcp/20006]} I1013 09:58:47.224372 1 loadbalancer.go:239] "Starting loadBalancer destinationRule controller" I1013 09:58:47.225230 1 config.go:317] "Starting service config controller" I1013 09:58:47.225253 1 shared_informer.go:240] Waiting for caches to sync for service config I1013 09:58:47.225332 1 tunnel.go:130] [MDNS] New stream between peer {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/127.0.0.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/169.254.96.16/tcp/20006 /ip4/10.5.0.2/tcp/20006]} success I1013 09:58:47.225409 1 shared_informer.go:240] Waiting for caches to sync for loadBalancer destinationRule I1013 09:58:47.225460 1 config.go:135] "Starting endpoints config controller" I1013 09:58:47.225470 1 shared_informer.go:240] Waiting for caches to sync for endpoints config I1013 09:58:47.225888 1 tunnel.go:118] [DHT] Discovery found peer: {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/10.5.0.2/tcp/20006 /ip4/192.168.1.42/tcp/20006 /ip4/127.0.0.1/tcp/20006]} I1013 09:58:47.226851 1 tunnel.go:130] [DHT] New stream between peer {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/10.5.0.2/tcp/20006 /ip4/192.168.1.42/tcp/20006 /ip4/127.0.0.1/tcp/20006]} success I1013 09:58:47.227979 1 tunnel.go:166] [MDNS] Discovery to default-worker : {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/127.0.0.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/10.244.2.1/tcp/20006 /ip4/169.254.96.16/tcp/20006 /ip4/10.5.0.2/tcp/20006]} I1013 09:58:47.228191 1 tunnel.go:166] [DHT] Discovery to default-worker : {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/10.5.0.2/tcp/20006 /ip4/192.168.1.42/tcp/20006 /ip4/127.0.0.1/tcp/20006]} I1013 09:58:47.297653 1 log.go:184] [INFO] 169.254.96.16:41729 - 63051 "HINFO IN 6778590575023996614.6041020207602238617. udp 57 false 512" NXDOMAIN qr,rd,ra 132 0.071851433s I1013 09:58:47.326254 1 shared_informer.go:247] Caches are synced for endpoints config I1013 09:58:47.326280 1 shared_informer.go:247] Caches are synced for loadBalancer destinationRule I1013 09:58:47.326280 1 shared_informer.go:247] Caches are synced for service config E1013 09:58:47.400588 1 proxier.go:552] "Failed to open portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" serviceName="vault/vault:vault" E1013 09:58:47.428342 1 proxier.go:552] "Failed to open portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" serviceName="vault/vault:vault-cluster" E1013 09:58:47.520382 1 proxier.go:552] "Failed to open portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" serviceName="metallb-system/nginx" E1013 09:58:47.587983 1 proxier.go:552] "Failed to open portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" serviceName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 09:58:47.609894 1 proxier.go:552] "Failed to open portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" serviceName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 09:58:47.640971 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 09:58:47.653816 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 09:58:47.675724 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 09:58:47.706793 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 09:58:47.742301 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 09:59:17.288585 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 09:59:17.337050 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 09:59:17.360974 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 09:59:17.389281 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 09:59:17.401219 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 09:59:47.467967 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 09:59:47.502568 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 09:59:47.534678 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 09:59:47.548799 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 09:59:47.574561 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" I1013 10:00:04.813229 1 tunnel.go:239] New stream between peer {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/127.0.0.1/tcp/20006 /ip4/10.5.0.2/tcp/20006 /ip4/192.168.1.42/tcp/20006]} success I1013 10:00:04.814221 1 loadbalancer.go:731] Dial libp2p network between vault-0 - {tcp default-worker 10.244.2.8:8200} E1013 10:00:04.825714 1 conn.go:50] "dest close failed" err="close tcp 169.254.96.16:43383->10.5.0.3:44956: use of closed network connection" I1013 10:00:04.827428 1 tunnel.go:239] New stream between peer {12D3KooWP6xc3WDcvWNT51M7vMQKvi4wtspKzMT2tEhgpmi7XjSw: [/ip4/127.0.0.1/tcp/20006 /ip4/10.5.0.2/tcp/20006 /ip4/192.168.1.42/tcp/20006]} success I1013 10:00:04.828104 1 loadbalancer.go:731] Dial libp2p network between vault-0 - {tcp default-worker 10.244.2.8:8200} E1013 10:00:17.637045 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:00:17.661533 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:00:17.709807 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:00:17.757266 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:00:17.835915 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:00:47.924114 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:00:47.951830 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:00:48.014957 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:00:48.091687 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:00:48.129515 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:01:18.236472 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:01:18.265873 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:01:18.305365 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:01:18.351177 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:01:18.403878 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:01:48.460204 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:01:48.479934 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:01:48.523876 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:01:48.555938 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:01:48.574154 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:02:18.640698 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:02:18.658518 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:02:18.733506 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:02:18.763687 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:02:18.795256 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:02:48.898800 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:02:48.925026 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:02:48.960802 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:02:49.050522 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:02:49.104452 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:03:19.155090 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:03:19.177999 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:03:19.202478 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:03:19.237281 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:03:19.252941 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:03:49.327365 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:03:49.356685 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:03:49.397488 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:03:49.456107 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:03:49.473749 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:04:19.628847 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:04:19.661769 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:04:19.687654 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:04:19.734482 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:04:19.803690 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:04:49.890310 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:04:49.904190 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:04:49.938686 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:04:49.954441 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:04:49.969685 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:05:05.107131 1 conn.go:53] "src close failed" err="close tcp 169.254.96.16:43383->10.5.0.3:44958: use of closed network connection" E1013 10:05:20.039136 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:05:20.071157 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:05:20.123358 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:05:20.191336 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:05:20.223720 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:05:50.324518 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:05:50.350999 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:05:50.373546 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:05:50.427216 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:05:50.461601 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:06:20.539964 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:06:20.550282 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:06:20.588061 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:06:20.659601 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:06:20.721304 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:06:50.779054 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:06:50.795026 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:06:50.816824 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:06:50.839194 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:06:50.863676 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:07:20.948636 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:07:20.989583 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:07:21.029421 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:07:21.072709 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:07:21.099430 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:07:51.181875 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:07:51.207924 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:07:51.228785 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:07:51.288683 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:07:51.303407 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:08:21.399899 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:08:21.417203 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:08:21.453454 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:08:21.464361 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:08:21.506771 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:08:51.565828 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:08:51.578406 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:08:51.599236 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:08:51.674830 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:08:51.737399 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:09:21.844037 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:09:21.872213 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:09:21.912162 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:09:21.947348 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:09:21.991153 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:09:52.093603 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:09:52.140492 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:09:52.163439 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:09:52.178210 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:09:52.205830 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:10:22.272642 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:10:22.284273 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:10:22.302388 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:10:22.349751 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:10:22.394018 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:10:52.535139 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:10:52.572575 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:10:52.600905 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:10:52.643240 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:10:52.739916 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:11:22.834322 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:11:22.860364 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:11:22.895299 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:11:22.956127 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:11:23.027497 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:11:53.105804 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:11:53.132401 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:11:53.154905 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:11:53.217808 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:11:53.265437 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:12:23.368707 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:12:23.418203 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:12:23.468089 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:12:23.539743 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:12:23.562857 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:12:53.671011 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:12:53.753152 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:12:53.798819 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:12:53.824971 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:12:53.861259 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:13:23.961799 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:13:24.001618 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:13:24.022462 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:13:24.034687 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:13:24.053759 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:13:54.174835 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:13:54.216191 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:13:54.260902 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:13:54.290132 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:13:54.315899 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:14:24.421499 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:14:24.474876 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:14:24.512446 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:14:24.537413 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:14:24.548979 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:14:54.663060 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:14:54.716955 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:14:54.757155 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:14:54.771847 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:14:54.814854 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:15:24.947856 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:15:24.986690 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:15:25.007823 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:15:25.028428 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:15:25.098305 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:15:55.168673 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:15:55.186640 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:15:55.201784 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:15:55.223056 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:15:55.265401 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:16:25.341514 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:16:25.359966 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:16:25.387134 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:16:25.435509 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:16:25.489835 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:16:55.567234 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:16:55.593092 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:16:55.642666 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:16:55.691897 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:16:55.763900 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:17:25.860859 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:17:25.884988 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:17:25.903411 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:17:25.916138 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:17:25.958941 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:17:56.037723 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:17:56.055968 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:17:56.091832 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:17:56.160260 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:17:56.187435 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:18:26.300837 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:18:26.336800 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:18:26.354395 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" E1013 10:18:26.365672 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:18:26.392402 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:18:56.515759 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30976/TCP: listen tcp :30976: bind: address already in use" servicePortName="vault/vault:vault-cluster" E1013 10:18:56.547715 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30694/TCP: listen tcp :30694: bind: address already in use" servicePortName="metallb-system/nginx" E1013 10:18:56.571014 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :30715/TCP: listen tcp :30715: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:https" E1013 10:18:56.651606 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :32227/TCP: listen tcp :32227: bind: address already in use" servicePortName="ingress-nginx/nginx-ingress-nginx-ingress:http" E1013 10:18:56.729074 1 proxier.go:422] "Failed to ensure portal" err="can't open node port for :31360/TCP: listen tcp :31360: bind: address already in use" servicePortName="vault/vault:vault" ``

Try yourself

If you want to spin up the same environment like i do, you can use our scripts.

Prerequisites:

devspace
mkcert
helm
docker
jq
kustomize

Create cluster and deploy everything

Run these commands:

$ git clone https://github.com/edgefarm/edgefarm.core.git 
$ cd edgefarm.core
$ git checkout edgemesh-rpi
$ devspace run create-kind-cluster
$ devspace run-pipeline init
$ devspace run-pipeline deploy-core
$ devspace run-pipeline deploy-edgemesh-cloud
$ devspace run-pipeline deploy-edgemesh-edge
$ dev/ports/forward.sh start
$ devspace run new-kubeedge-vault-device clownfish

Build custom edgecore

We have a pending PR in kubeedge, where we integrate Hasicorp vault to kubeedge to provide certiicates that are handeled by vault.

$ git clone https://github.com/edgefarm/kubeedge
$ git checkout vault_on_1.12.0
$ make crossbuild WHAT=edgecore ARM_VERSION=GOARM8

Provision your Raspberry Pi device

Make sure, that there are no residues from an older edgecore instllation left (edgecore stopped, rm /var/lib/kubeedge/edgecore.db, docker kill $(docker ps -aq) && docker rm $(docker ps -aq))

$ scp _output/local/bin/edgecore <yourdevice>:/usr/local/bin/edgecore
$ kubectl get secrets -n edge-nodes device-vault-token-clownfish -o jsonpath='{.data.token}' | base64 -d | xargs echo > token
$ scp token  <yourdevice>:/etc/kubeedge/vault/token

Use this config and replace all occurances of '192-168-1-42.nip.io` with your local machines nip.io address.

$ DEFAULT_INTERFACE=$(route | grep default | awk '{print $8}')
$ ip -4 -br addr show | grep ${DEFAULT_INTERFACE} | awk '{print $3}' | cut -d '/' -f1 | sed -e 's/\./-/g' | xargs -I {} echo {}.nip.io

edgecore.yaml

apiVersion: edgecore.config.kubeedge.io/v1alpha1
database:
  aliasName: default
  dataSource: /var/lib/kubeedge/edgecore.db
  driverName: sqlite3
kind: EdgeCore
modules:
  dbTest:
    enable: false
  deviceTwin:
    enable: true
  edgeHub:
    enable: true
    heartbeat: 15
    httpServer: https://192-168-1-42.nip.io:10002
    projectID: e632aba927ea4ac2b575ec1603d56f10
    quic:
      enable: false
    rotateCertificates: true
    tlsCaFile: /etc/kubeedge/certs/rootCA.pem
    tlsCertFile: /etc/kubeedge/certs/node.pem
    tlsPrivateKeyFile: /etc/kubeedge/certs/node.key
    websocket:
      enable: true
      handshakeTimeout: 30
      readDeadline: 15
      server: 192-168-1-42.nip.io:10000
      writeDeadline: 15
    vault:
      # enable vault integration
      enable: true
      tokenFile: /etc/kubeedge/vault/token
      authrole: kubeedge-client
      # the vault role to use
      role: client
      # the subject name to requset
      commonName: clownfish.edgefarm.io
      # the requested validity period of the certificate
      ttl: 1h
      # The address of the vault server
      # (the tlsCaFile is used to validate the server certificate )
      address: "https://192-168-1-42.nip.io:8200"
      serverCA: /etc/kubeedge/vault/ca.crt
      PKI: kubeedge
  edgeStream:
    enable: true
    handshakeTimeout: 30
    readDeadline: 15
    server: 192-168-1-42.nip.io:10004
    tlsTunnelCAFile: /etc/kubeedge/certs/rootCA.pem
    tlsTunnelCertFile: /etc/kubeedge/certs/node.pem
    tlsTunnelPrivateKeyFile: /etc/kubeedge/certs/node.key
    writeDeadline: 15
  edged:
    enable: true
    hostnameOverride: clownfish
    taints:
      - key: edgefarm.applications
        effect: NoExecute
    cgroupDriver: systemd
    cgroupRoot: ""
    cgroupsPerQOS: true
    clusterDNS: 169.254.96.16
    clusterDomain: cluster.local
    cniBinDir: /opt/cni/bin
    cniCacheDirs: /var/lib/cni/cache
    cniConfDir: /etc/cni/net.d
    concurrentConsumers: 5
    devicePluginEnabled: false
    dockerAddress: unix:///var/run/docker.sock
    edgedMemoryCapacity: 7852396000
    enableMetrics: true
    gpuPluginEnabled: false
    imageGCHighThreshold: 80
    imageGCLowThreshold: 40
    imagePullProgressDeadline: 60
    maximumDeadContainersPerPod: 1
    networkPluginMTU: 1500
    nodeStatusUpdateFrequency: 10
    podSandboxImage: kubeedge/pause:3.1
    registerNode: true
    registerNodeNamespace: default
    remoteImageEndpoint: unix:///var/run/dockershim.sock
    remoteRuntimeEndpoint: unix:///var/run/dockershim.sock
    runtimeRequestTimeout: 2
    runtimeType: docker
    volumeStatsAggPeriod: 60000000000
  eventBus:
    enable: true
    eventBusTLS:
      enable: false
    mqttMode: 0
    mqttQOS: 0
    mqttRetain: false
    mqttServerInternal: tcp://127.0.0.1:1884
    mqttSessionQueueSize: 100
  metaManager:
    enable: true
    contextSendGroup: hub
    contextSendModule: websocket
    metaServer:
      debug: false
      enable: true
    podStatusSyncInterval: 60
    remoteQueryTimeout: 60
  serviceBus:
    enable: false

That's it. Your device should register at cloudcore and the new node can be used.

Oct 13 '22 10:10 siredmar

kubectl label services -n kube-system kube-dns service.edgemesh.kubeedge.io/service-proxy-name-

If you enable the cacheDNS, then you must proxy kube-dns.

Oct 14 '22 01:10 Poorunga

@Poorunga is there any note in the docs? It seems to me that there are plenty of obstacles that might seem obvious to solve to some people, but aren't at all for others.

Oct 14 '22 05:10 siredmar

No, the documentation has no corresponding instructions. You're right, I'll add the documentation later.

Oct 17 '22 15:10 Poorunga

@Poorunga would you please explain what the cacheDNS functionality does?

If the edge node is offline (bad LTE connection), is the edge DNS also able to resolve to node local services that are deployed with an kind edgeApplication?

Oct 25 '22 06:10 siredmar

cacheDNS requires you to specify the upstream dns nameserver yourself, for example:

10.96.0.10    --> coredns cluster ip
100.125.3.250 --> your custom dns

When edgemesh's dns runs in cacheDNS mode, all domain name requests in pods will go through the following paths: 169.254.96.16:53 -> coredns forward plugin(in cacheDNS) -> 10.96.10.10 or 100.125.3.250

If your edge node is offline, edge node cannot access 10.96.0.10(coredns in cloud) but can access 100.125.3.250(a dns in same LAN) then cacheDNS can still work normally.

Oct 25 '22 06:10 Poorunga

So basically cacheDNS is just a setting that allows to configure some fallback DNS servers in case the clusterDNS isn't reachable? So in my case not needed, right?

Oct 25 '22 06:10 siredmar

So basically cacheDNS is just a setting that allows to configure some fallback DNS servers in case the clusterDNS isn't reachable? So in my case not needed, right?

You are right.

BTW, you can exec to edgemesh-agent pod in edgenode, then cat /Corefile see some info.

Oct 25 '22 07:10 Poorunga

Thanks for the support!

Oct 25 '22 07:10 siredmar