bug: apisix-etcd pod CrashLoopBackOff

[Asakaigg]

Current Behavior

I am using apache apisix on the kubernetes but when I have installed the apisix but it got the error like this

Expected Behavior

I need install the apisix on the kubernetes but when I move port forwarding two pods are stopped

Error Logs

No response

Steps to Reproduce

helm install helm install bitnami helm install apisix apisix/apisix --namespace apache-gateway

Environment

apisix version: apisix-3.0.2 Operating system: Ubuntu 22.0.4

[hanqingwu]

Can you check the reason about why etcd apisix-etcd-0 not running ?

[Asakaigg]

this is the guide from apisix due to this error, I have reinstalled with this guide but all pods is not running https://apisix.apache.org/blog/2021/12/15/deploy-apisix-in-kubernetes/ using kubectl describe pod pod_name --n namespace_name , I got this error using kubectl logs pod_name --n namespace_name, I got this error

[Asakaigg]

Can you check the reason about why etcd apisix-etcd-0 not running ? Could you please check the errors ? It is a little challenge from me. I hope you help me on this

[hanqingwu]

Yes, the etcd cluster is not running normal. So you can scale etcd replicas from 3 to 1 . Or you can try preference etcd cluster config like this:

      ETCD_ENABLE_V2:                    true
      ALLOW_NONE_AUTHENTICATION:         yes
      ETCD_NAME:                         cooper-apisix-etcd-1 (v1:metadata.name)
      POD_IP:                             (v1:status.podIP)
      ETCD_ADVERTISE_CLIENT_URLS:        http://$(POD_IP):2379
      ETCD_INITIAL_ADVERTISE_PEER_URLS:  http://$(ETCD_NAME).cooper-apisix-etcd.paas-middleware-workspace-middleware.svc:2380
      ETCD_LISTEN_CLIENT_URLS:           http://0.0.0.0:2379
      ETCD_LISTEN_PEER_URLS:             http://0.0.0.0:2380
      ETCD_INITIAL_CLUSTER:              cooper-apisix-etcd-0=http://cooper-apisix-etcd-0.cooper-apisix-etcd.paas-middleware-workspace-middleware.svc:2380,cooper-apisix-etcd-1=http://cooper-apisix-etcd-1.cooper-apisix-etcd.paas-middleware-workspace-middleware.svc:2380,cooper-apisix-etcd-2=http://cooper-apisix-etcd-2.cooper-apisix-etcd.paas-middleware-workspace-middleware.svc:2380
      ETCD_INITIAL_CLUSTER_TOKEN:        apisix-etcd-cluster
      ETCD_INITIAL_CLUSTER_STATE:        new
      ETCD_DEBUG:                        true

[Asakaigg]

ETCD_ADVERTISE_CLIENT_URLS

Yes, the etcd cluster is not running normal. So you can scale etcd replicas from 3 to 1 . Or you can try preference etcd cluster config like this:

      ETCD_ENABLE_V2:                    true
      ALLOW_NONE_AUTHENTICATION:         yes
      ETCD_NAME:                         cooper-apisix-etcd-1 (v1:metadata.name)
      POD_IP:                             (v1:status.podIP)
      ETCD_ADVERTISE_CLIENT_URLS:        http://$(POD_IP):2379
      ETCD_INITIAL_ADVERTISE_PEER_URLS:  http://$(ETCD_NAME).cooper-apisix-etcd.paas-middleware-workspace-middleware.svc:2380
      ETCD_LISTEN_CLIENT_URLS:           http://0.0.0.0:2379
      ETCD_LISTEN_PEER_URLS:             http://0.0.0.0:2380
      ETCD_INITIAL_CLUSTER:              cooper-apisix-etcd-0=http://cooper-apisix-etcd-0.cooper-apisix-etcd.paas-middleware-workspace-middleware.svc:2380,cooper-apisix-etcd-1=http://cooper-apisix-etcd-1.cooper-apisix-etcd.paas-middleware-workspace-middleware.svc:2380,cooper-apisix-etcd-2=http://cooper-apisix-etcd-2.cooper-apisix-etcd.paas-middleware-workspace-middleware.svc:2380
      ETCD_INITIAL_CLUSTER_TOKEN:        apisix-etcd-cluster
      ETCD_INITIAL_CLUSTER_STATE:        new
      ETCD_DEBUG:                        true

Could you explain the difference between standard and your configuration ?

git clone https://github.com/zaunist/apisix-on-kubernetes.git I have download the etcd.yaml and changed

• name: ETCD_ADVERTISE_CLIENT_URLS value: >- http://$(MY_POD_IP):2379 like this for other configuration, I am not sure it works on my side Because when I check ETCD_DEBUG and ETCD_ENABLE_V2, it didn't work

[Asakaigg]

this is my pod yaml using kubectl edit command for the pod but it seems different with your config

• name: etcd image: docker.io/bitnami/etcd:3.5.7-debian-11-r14 ports: - name: client containerPort: 2379 protocol: TCP - name: peer containerPort: 2380 protocol: TCP env: - name: BITNAMI_DEBUG value: 'false' - name: MY_POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: MY_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: MY_STS_NAME value: apisix-etcd - name: ETCDCTL_API value: '3' - name: ETCD_ON_K8S value: 'yes' - name: ETCD_START_FROM_SNAPSHOT value: 'no' - name: ETCD_DISASTER_RECOVERY value: 'no' - name: ETCD_NAME value: $(MY_POD_NAME) - name: ETCD_DATA_DIR value: /bitnami/etcd/data - name: ETCD_LOG_LEVEL value: info - name: ALLOW_NONE_AUTHENTICATION value: 'yes' - name: ETCD_AUTH_TOKEN value: >- jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m - name: ETCD_ADVERTISE_CLIENT_URLS value: >- http://$(MY_POD_NAME).apisix-etcd-headless.api-gateway.svc.cluster.local:2379,http://apisix-etcd.api-gateway.svc.cluster.local:2379 - name: ETCD_LISTEN_CLIENT_URLS value: http://0.0.0.0:2379 - name: ETCD_INITIAL_ADVERTISE_PEER_URLS value: >- http://$(MY_POD_NAME).apisix-etcd-headless.api-gateway.svc.cluster.local:2380 - name: ETCD_LISTEN_PEER_URLS value: http://0.0.0.0:2380 - name: ETCD_INITIAL_CLUSTER_TOKEN value: etcd-cluster-k8s - name: ETCD_INITIAL_CLUSTER_STATE value: new - name: ETCD_INITIAL_CLUSTER value: >- apisix-etcd-0=http://apisix-etcd-0.apisix-etcd-headless.api-gateway.svc.cluster.local:2380,apisix-etcd-1=http://apisix-etcd-1.apisix-etcd-headless.api-gateway.svc.cluster.local:2380,apisix-etcd-2=http://apisix-etcd-2.apisix-etcd-headless.api-gateway.svc.cluster.local:2380 - name: ETCD_CLUSTER_DOMAIN value: apisix-etcd-headless.api-gateway.svc.cluster.local

[pietrogoddibit2win]

Same issue on GKE