redpanda icon indicating copy to clipboard operation
redpanda copied to clipboard

`Address already in use` error when starting cluster with external connectivity enabled

Open vuldin opened this issue 4 years ago • 9 comments

Version & Environment

macOS v11.6 (20G165) docker v4.3.0 (engine v20.10.8) minikube v1.24.0 kubernetes v1.22.3 kubectl v1.22.3 cert-manager v1.6.1 redpanda v21.10.2

What went wrong?

I ran into issues where the pod never initializes while trying to add external connectivity to my local redpanda cluster:

> kubectl get pods -n redpanda-test  
NAME                 READY   STATUS     RESTARTS   AGE
redpanda-cluster-0   0/2     Init:0/1   0          15m

There are multiple warning events shown with the following command kubectl get events -n redpanda-test:

...
15s         Warning   FailedCreatePodSandBox   pod/redpanda-cluster-0                             Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_f2ed69ae-748b-46d0-91e1-d1e1781b4d0f_0 (303393cb8d34b8502e6ca44b29c642279709dc06a70e7edfb801996b539ad411): Error starting userland proxy: listen tcp4 0.0.0.0:30192: bind: address already in use
3s          Normal    SandboxChanged           pod/redpanda-cluster-0                             Pod sandbox changed, it will be killed and re-created.
14s         Warning   FailedCreatePodSandBox   pod/redpanda-cluster-0                             Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_f2ed69ae-748b-46d0-91e1-d1e1781b4d0f_1 (ad14646941da4b62f156b129a795b5c8f1fecce54e7e078f139b5b5a554472ca): Error starting userland proxy: listen tcp4 0.0.0.0:30192: bind: address already in use
13s         Warning   FailedCreatePodSandBox   pod/redpanda-cluster-0                             Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_f2ed69ae-748b-46d0-91e1-d1e1781b4d0f_2 (a470c0039f5ab1fb66ee2980acdf99d64492b4f6d9aca32c713765de32905eee): Error starting userland proxy: listen tcp4 0.0.0.0:30192: bind: address already in use
12s         Warning   FailedCreatePodSandBox   pod/redpanda-cluster-0                             Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_f2ed69ae-748b-46d0-91e1-d1e1781b4d0f_3 (d078fe9b7c803865bc4b4530aa45a01ce73307faacbabef07de4ee8cfd7a745b): Error starting userland proxy: listen tcp4 0.0.0.0:30192: bind: address already in use
11s         Warning   FailedCreatePodSandBox   pod/redpanda-cluster-0                             Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_f2ed69ae-748b-46d0-91e1-d1e1781b4d0f_4 (3b686dae3229ea66782f6c7cf7d6fa18b7cb57b63081477ad89cd1a79943de08): Error starting userland proxy: listen tcp4 0.0.0.0:30192: bind: address already in use
10s         Warning   FailedCreatePodSandBox   pod/redpanda-cluster-0                             Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_f2ed69ae-748b-46d0-91e1-d1e1781b4d0f_5 (4adce86189d20f1f6dbea8227471f52f1aa8e24fc1cd978c1998aeac4be33f01): Error starting userland proxy: listen tcp4 0.0.0.0:31864: bind: address already in use
9s          Warning   FailedCreatePodSandBox   pod/redpanda-cluster-0                             Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_f2ed69ae-748b-46d0-91e1-d1e1781b4d0f_6 (d51fa664f9dfab70de748babe668e7df2763b26ae3c292288f838271306069ce): Error starting userland proxy: listen tcp4 0.0.0.0:30192: bind: address already in use
8s          Warning   FailedCreatePodSandBox   pod/redpanda-cluster-0                             Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_f2ed69ae-748b-46d0-91e1-d1e1781b4d0f_7 (59846fe2a7312b12d818b0bb12ad2ae0c57e93840a96465988aa28bd36dfd4e6): Error starting userland proxy: listen tcp4 0.0.0.0:30192: bind: address already in use
7s          Warning   FailedCreatePodSandBox   pod/redpanda-cluster-0                             Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_f2ed69ae-748b-46d0-91e1-d1e1781b4d0f_8 (1c401b2540df89a9abab668508996546334f0119d4e00a975df1cc29499bbc68): Error starting userland proxy: listen tcp4 0.0.0.0:30192: bind: address already in use
3s          Warning   FailedCreatePodSandBox   pod/redpanda-cluster-0                             (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_f2ed69ae-748b-46d0-91e1-d1e1781b4d0f_12 (fff412d179c3f474b638f33e2117953022017124a469571b6204ef99ee38e5cb): Error starting userland proxy: listen tcp4 0.0.0.0:30192: bind: address already in use
...

What should have happened instead?

The sandbox container should have been successfully created.

How to reproduce the issue?

I'm using the following cluster config (named redpanda.yml in the commands below):

apiVersion: redpanda.vectorized.io/v1alpha1
kind: Cluster
metadata:
  name: redpanda-cluster
spec:
  image: 'vectorized/redpanda'
  version: 'latest'
  replicas: 1
  resources:
    requests:
      cpu: 1
      memory: 1.2Gi
    limits:
      cpu: 1
      memory: 1.2Gi
  configuration:
    rpcServer:
      port: 33145
    kafkaApi:
      - port: 9092
      - external:
          enabled: true
    pandaproxyApi:
      - port: 8082
      - external:
          enabled: true
    adminApi:
      - port: 9644
      - external:
          enabled: true
    developerMode: true
    autoCreateTopics: false
  1. Install pre-reqs and setup helm repos (details here: https://vectorized.io/docs/quick-start-kubernetes/)
  2. minikube start
  3. export REDPANDA_VERSION=$(curl -s https://api.github.com/repos/vectorizedio/redpanda/releases/latest | jq -r .tag_name)
  4. kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.yaml
  5. kubectl get pods -w --namespace cert-manager
  6. noglob kubectl apply -k https://github.com/vectorizedio/redpanda/src/go/k8s/config/crd\?ref\=$REDPANDA_VERSION
  7. helm install --namespace redpanda-system --create-namespace redpanda-system --version $REDPANDA_VERSION redpanda/redpanda-operator
  8. kubectl create ns redpanda-test
  9. kubectl apply -n redpanda-test -f redpanda.yml

Then run the following command to see the error events above:

kubectl get events -n redpanda-test

Additional information

Please attach any relevant logs, backtraces, or metric charts.

JIRA Link: CORE-792

vuldin avatar Dec 04 '21 18:12 vuldin

I just realized the event logs remain after deleting the cluster with:

kubectl delete -n redpanda-test -f redpanda.yml

So I'm not sure that the event messages above are related to why the redpanda cluster pod never makes it out of Init status.

vuldin avatar Dec 04 '21 18:12 vuldin

It looks like the events are related to why the pod isn't initializing properly after running the following commands:

> kubectl -n redpanda-test logs redpanda-cluster-0 redpanda
Error from server (BadRequest): container "redpanda" in pod "redpanda-cluster-0" is waiting to start: PodInitializing
> kubectl -n redpanda-test logs redpanda-cluster-0 rpk-status
Error from server (BadRequest): container "rpk-status" in pod "redpanda-cluster-0" is waiting to start: PodInitializing
> kubectl -n redpanda-test logs redpanda-cluster-0 redpanda-configurator
Error from server (BadRequest): container "redpanda-configurator" in pod "redpanda-cluster-0" is waiting to start: PodInitializing

And:

> kubectl -n redpanda-test describe pod redpanda-cluster-0                 
Name:           redpanda-cluster-0
Namespace:      redpanda-test
Priority:       0
Node:           minikube/192.168.49.2
Start Time:     Sat, 04 Dec 2021 13:21:55 -0500
Labels:         app.kubernetes.io/component=redpanda
                app.kubernetes.io/instance=redpanda-cluster
                app.kubernetes.io/name=redpanda
                controller-revision-hash=redpanda-cluster-c6b4d755f
                statefulset.kubernetes.io/pod-name=redpanda-cluster-0
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  StatefulSet/redpanda-cluster
Init Containers:
  redpanda-configurator:
    Container ID:   
    Image:          vectorized/configurator:v21.10.2
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  1288490188800m
    Requests:
      cpu:     1
      memory:  1288490188800m
    Environment:
      SERVICE_FQDN:                     redpanda-cluster.redpanda-test.svc.cluster.local.
      CONFIG_SOURCE_DIR:                /mnt/operator
      CONFIG_DESTINATION:               /etc/redpanda/redpanda.yaml
      REDPANDA_RPC_PORT:                33145
      NODE_NAME:                         (v1:spec.nodeName)
      EXTERNAL_CONNECTIVITY:            true
      EXTERNAL_CONNECTIVITY_SUBDOMAIN:  
      HOST_PORT:                        31974
      PROXY_HOST_PORT:                  31714
    Mounts:
      /etc/redpanda from config-dir (rw)
      /mnt/operator from configmap-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hdt87 (ro)
Containers:
  redpanda:
    Container ID:  
    Image:         vectorized/redpanda:latest
    Image ID:      
    Ports:         33145/TCP, 9644/TCP, 9092/TCP, 8082/TCP, 9093/TCP, 9645/TCP, 8083/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP, 31974/TCP, 32732/TCP, 31714/TCP
    Command:
      /usr/bin/rpk
    Args:
      redpanda
      start
      --check=false
      --advertise-rpc-addr=$(POD_NAME).redpanda-cluster.redpanda-test.svc.cluster.local.:33145
      --overprovisioned
      --kernel-page-cache=true
      --default-log-level=debug
      --smp=1
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  1288490188800m
    Requests:
      cpu:     1
      memory:  1288490188800m
    Environment:
      REDPANDA_ENVIRONMENT:  kubernetes
      POD_NAME:              redpanda-cluster-0 (v1:metadata.name)
      POD_NAMESPACE:         redpanda-test (v1:metadata.namespace)
      POD_IP:                 (v1:status.podIP)
    Mounts:
      /etc/redpanda from config-dir (rw)
      /var/lib/redpanda/data from datadir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hdt87 (ro)
  rpk-status:
    Container ID:  
    Image:         vectorized/redpanda:latest
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/local/bin/rpk-status.sh
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  10M
    Requests:
      cpu:     200m
      memory:  10M
    Environment:
      REDPANDA_ENVIRONMENT:  kubernetes
    Mounts:
      /etc/redpanda from config-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hdt87 (ro)
Conditions:
  Type              Status
  Initialized       False 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  datadir:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  datadir-redpanda-cluster-0
    ReadOnly:   false
  configmap-dir:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      redpanda-cluster-base
    Optional:  false
  config-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-hdt87:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                     From               Message
  ----     ------                  ----                    ----               -------
  Normal   Scheduled               3m55s                   default-scheduler  Successfully assigned redpanda-test/redpanda-cluster-0 to minikube
  Warning  FailedCreatePodSandBox  3m55s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_13b4dfcd-e649-4d4f-8db1-e3008a6c1225_0 (a0bf63912b34f3042a32610d9dcfabf8b5597e8cebd7ee63fbc97a9f3a63b71e): Error starting userland proxy: listen tcp4 0.0.0.0:32732: bind: address already in use
  Warning  FailedCreatePodSandBox  3m54s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_13b4dfcd-e649-4d4f-8db1-e3008a6c1225_1 (1dc01a9195f46d79c9db629ca927d17dc4f63ebfb8de9a257edba111f31cd5c5): Error starting userland proxy: listen tcp4 0.0.0.0:32732: bind: address already in use
  Warning  FailedCreatePodSandBox  3m53s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_13b4dfcd-e649-4d4f-8db1-e3008a6c1225_2 (c2bd4bbd285e4f7c3191be46793d6e1fadbbdfc4cc6c9cbc3e6f74edbb24aa67): Error starting userland proxy: listen tcp4 0.0.0.0:32732: bind: address already in use
  Warning  FailedCreatePodSandBox  3m52s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_13b4dfcd-e649-4d4f-8db1-e3008a6c1225_3 (93315df7131e0253d26b6cbd9accb7500b10d1acb8c5695032ba3522788d528e): Error starting userland proxy: listen tcp4 0.0.0.0:32732: bind: address already in use
  Warning  FailedCreatePodSandBox  3m51s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_13b4dfcd-e649-4d4f-8db1-e3008a6c1225_4 (9608eaaa73241ad284c7ac0642ec65235005df31f53731a4998241f013ea9223): Error starting userland proxy: listen tcp4 0.0.0.0:32732: bind: address already in use
  Warning  FailedCreatePodSandBox  3m50s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_13b4dfcd-e649-4d4f-8db1-e3008a6c1225_5 (ff51cdefd41c0443bfb5441b38a68c9b7078b9106031df455f257954a8361c11): Error starting userland proxy: listen tcp4 0.0.0.0:32732: bind: address already in use
  Warning  FailedCreatePodSandBox  3m49s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_13b4dfcd-e649-4d4f-8db1-e3008a6c1225_6 (99ebe453837f8eb1a69ed9ff0a7d0b2efb30b8c1ee7c84a735bae0f10b86826f): Error starting userland proxy: listen tcp4 0.0.0.0:32732: bind: address already in use
  Warning  FailedCreatePodSandBox  3m48s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_13b4dfcd-e649-4d4f-8db1-e3008a6c1225_7 (ae0210d8b8a7c151b6ec3f387d65d358dca1218bca342fdd8d1e7bce1a0a9465): Error starting userland proxy: listen tcp4 0.0.0.0:32732: bind: address already in use
  Warning  FailedCreatePodSandBox  3m47s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_13b4dfcd-e649-4d4f-8db1-e3008a6c1225_8 (05be43fd021be5827e4ec683b0b697fa731114c31307818163cb8ff932b375b9): Error starting userland proxy: listen tcp4 0.0.0.0:32732: bind: address already in use
  Normal   SandboxChanged          3m43s (x12 over 3m55s)  kubelet            Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  3m43s (x4 over 3m46s)   kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "redpanda-cluster-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_13b4dfcd-e649-4d4f-8db1-e3008a6c1225_12 (94ea72394c6a224d8734f4e6a31addb08ef58df32a91250bf997b921ec95b041): Error starting userland proxy: listen tcp4 0.0.0.0:32732: bind: address already in use

vuldin avatar Dec 04 '21 18:12 vuldin

I removed the "external enabled true" yaml lines from all but kafkaApi, and then added a port:

    kafkaApi:
      - port: 9092
      - external:
          enabled: true
        port: 30001

When applying this a different error (from the usual validation error) appears:

The Cluster "redpanda-cluster" is invalid: spec.configuration.kafkaApi: Invalid value: []v1alpha1.KafkaAPI{v1alpha1.KafkaAPI{Port:9092, External:v1alpha1.ExternalConnectivityConfig{Enabled:false, Subdomain:""}, TLS:v1alpha1.KafkaAPITLS{Enabled:false, IssuerRef:(*v1.ObjectReference)(nil), NodeSecretRef:(*v1.ObjectReference)(nil), RequireClientAuth:false}}, v1alpha1.KafkaAPI{Port:30001, External:v1alpha1.ExternalConnectivityConfig{Enabled:true, Subdomain:""}, TLS:v1alpha1.KafkaAPITLS{Enabled:false, IssuerRef:(*v1.ObjectReference)(nil), NodeSecretRef:(*v1.ObjectReference)(nil), RequireClientAuth:false}}}: external kafka api listener cannot have port specified, it's autogenerated

I thought specifying the port would be the way to go since I found a test that does this: https://github.com/vectorizedio/redpanda/blob/91ae813c79e5754808489a30769b0187efc75e86/src/go/k8s/tests/e2e/external-connectivity/00-external-connectivity.yaml#L23-L26

And I see this in result.KafkaAPI.ExternalPortIsGenerated file (set to true): https://github.com/vectorizedio/redpanda/blob/91ae813c79e5754808489a30769b0187efc75e86/src/go/k8s/pkg/networking/ports.go#L58

But I don't see an associated field in the yaml file: https://github.com/vectorizedio/redpanda/blob/d4c859416e9d65df737f764ae63760c183547e47/src/go/k8s/config/crd/bases/redpanda.vectorized.io_clusters.yaml

vuldin avatar Dec 04 '21 21:12 vuldin

I looked into what is taking the port in the minikube VM. I think the issue is that kube-proxy is listening on the port that redpanda is trying to allocate for external kafka API connectivity. I deleted the minikube cluster went through all the steps listed above, and this time the port trying to be used was 31620:

Dec 05 02:19:33 minikube dockerd[380]: time="2021-12-05T02:19:33.878791900Z" level=warning msg="Failed to allocate and map port 31620-31620: Error starting userland proxy: listen tcp4 0.0.0.0:31620: bind: address already in use"
Dec 05 02:19:33 minikube dockerd[380]: time="2021-12-05T02:19:33.914869200Z" level=error msg="2bddea84da720f09a4e4fa50fefe543c32b7e5fa21229dd3634308a73f7ec3a6 cleanup: failed to delete container from containerd: no such container"
Dec 05 02:19:33 minikube dockerd[380]: time="2021-12-05T02:19:33.914951200Z" level=error msg="Handler for POST /v1.41/containers/2bddea84da720f09a4e4fa50fefe543c32b7e5fa21229dd3634308a73f7ec3a6/start returned error: driver failed programming external connectivity on endpoint k8s_POD_redpanda-cluster-0_redpanda-test_0d842566-c03f-4234-be55-761c63903a5e_22 (edc8ea3f53a8846347ac94a8ce10ef773204afed30dc5e5c91bf1d63d13f5b93): Error starting userland proxy: listen tcp4 0.0.0.0:31620: bind: address already in use"

I connected to the minikube VM, installed net-tools, and ran the following commands:

root@minikube:/home/docker# netstat -tulpn | grep 31620
tcp        0      0 0.0.0.0:31620           0.0.0.0:*               LISTEN      2794/kube-proxy     
root@minikube:/home/docker# ls -l /proc/2794/exe
lrwxrwxrwx 1 root root 0 Dec  5 02:06 /proc/2794/exe -> /usr/local/bin/kube-proxy
root@minikube:/home/docker# ps -ef | grep kube-proxy
root        2794    2745  0 02:06 ?        00:00:01 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf --hostname-override=minikube
root       15585    9572  0 02:24 pts/1    00:00:00 grep --color=auto kube-proxy

minikube.log

vuldin avatar Dec 05 '21 02:12 vuldin

Hit the same issue as well.

docker version:   20.10.12
minikube version: v1.25.1
redpanda version: v21.11.4

Got the following error in Pod events

  Warning  FailedCreatePodSandBox  16m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "red-0": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_red-0_redpanda_fa4f7c2b-f87e-4118-8593-1819b0bd84e5_0 (6e12921ed23883060335af2d3c1f003fe00d58e96dcd2cf49944c28fd708596b): Error starting userland proxy: listen tcp4 0.0.0.0:30001: bind: address already in use

Where meaningful part is

Error starting userland proxy: listen tcp4 0.0.0.0:30001: bind: address already in use

and taking a look into node with

minikube ssh

it is indeed in use by kube-proxy

tcp        0      0 0.0.0.0:30001           0.0.0.0:*               LISTEN      2354/kube-proxy     

sunsingerus avatar Feb 11 '22 12:02 sunsingerus

The only difference with provided example is external

    - external:
        enabled: true
        subdomain: local.rp
      port: 30001

With the whole manifest is like this:

apiVersion: redpanda.vectorized.io/v1alpha1
kind: Cluster
metadata:
  name: red
spec:
  image: "vectorized/redpanda"
  version: "latest"
  replicas: 1
  resources:
    requests:
      cpu: 1
      memory: 1.2Gi
    limits:
      cpu: 1
      memory: 1.2Gi
  configuration:
    rpcServer:
      port: 33145
    kafkaApi:
    - port: 9092
    - external:
        enabled: true
        subdomain: local.rp
      port: 30001
    pandaproxyApi:
    - port: 8082
    adminApi:
    - port: 9644
    developerMode: true

sunsingerus avatar Feb 11 '22 12:02 sunsingerus

Without external section all is fine, cluster is created successfully

sunsingerus avatar Feb 11 '22 12:02 sunsingerus

Errors mentioned in this issue should can manifest on minicube as Service type NodePort and the POD specification hostPort is pointing to the same port. The minikube implementation might be different than kind cluster. For local development I suggest to use https://github.com/redpanda-data/redpanda/blob/dev/src/go/k8s/docs/external-connectivity-localhost.md.

Another option would be to use helm from our official docs. https://docs.redpanda.com/docs/platform/quickstart/kubernetes-qs-dev/#use-helm-to-install-redpanda

RafalKorepta avatar Nov 14 '22 17:11 RafalKorepta

This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.

github-actions[bot] avatar Jun 05 '25 06:06 github-actions[bot]

This issue was closed due to lack of activity. Feel free to reopen if it's still relevant.

github-actions[bot] avatar Jun 20 '25 06:06 github-actions[bot]