[bitnami/apisix] mtls connection to etcd failing after upgrading

[raji032]

Name and Version

bitnami/apisix:3.7.0-debian-11-r3

What architecture are you using?

None

What steps will reproduce the bug?

1. Install Bitnami etcd cluster that uses mtls connection
2. Install apisix helmchart with controlPlane.enabled=true
3. Update etcd configurations in defaultconfig. 4

etcd:
        host:
          - "https://apisix-etcd.hostname:443"
        mtls:
          key_file: /bitnami/certs/{{ .Values.controlPlane.tls.certKeyFilename }}
          cert_file: /bitnami/certs/{{ .Values.controlPlane.tls.certFilename }}
          ca_file: /bitnami/certs/{{ .Values.controlPlane.tls.certCAFilename }}
        prefix: /apisix
        timeout: 30
        use_grpc: false
        startup_retry: 60
        user: root
        password: password

5. Same configuration worked with version 3.5.0-debian-11-r0 but failing with latest version.

What is the expected behavior?

Pass client certificate with request and connect successfully to ETCD

What do you see instead?

 SSL_do_handshake() failed (SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:SSL alert number 42), context: ngx.timer
[lua] config_etcd.lua:159: etcd get: has no healthy etcd endpoint available, context: ngx.timer

Additional information

1. Also tried with below apisix configurations.

 etcd:
        host:
          - "https://apisix-etcd.hostname:443"
        tls:
          key: /bitnami/certs/{{ .Values.controlPlane.tls.certKeyFilename }}
          cert: /bitnami/certs/{{ .Values.controlPlane.tls.certFilename }}
        prefix: /apisix
        timeout: 30
        use_grpc: false
        startup_retry: 60
        user: root
        password: password

2. I am using bitnami helmchart to install apisix.

[joancafom]

Hi @raji032

How are you installing the ETCD chart in your cluster? What are the configuration params you are using? As pero your steps, you seem to be installing ETCD first as a standalone chart instead of a subchart. Are you referring to this installation correctly using the externalEtcd param in apisix's values.yaml?

[raji032]

Hi, Yes, I am using externalEtcd to configure ETCD details. values.yaml default-config section does not have parameters to add etcd.tls so even when securetransport is set as true, the deployment fails at 'wait for-etcd' init container as the client certs are not being sent for mtls connection.

Error in wait-for-etcd container

---

curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.

---

Here is the command in the container-

check_etcd() {
              local -r etcd_host="${1:-?missing etcd}"
              if curl --max-time 5 "${etcd_host}/version" | grep etcdcluster; then
                 return 0
              else
                 return 1
              fi
          }

After I patch the deployment file with client certs in the curl command, it proceeds to next step and fails at prepare-apisix container.

check_etcd() {
              local -r etcd_host="${1:-?missing etcd}"
              if curl --cacert /bitnami/certs/ca.crt --cert /bitnami/certs/tls.crt --key /bitnami/certs/tls.key --max-time 5 "${etcd_host}/version" | grep etcdcluster; then
                 return 0
              else
                 return 1
              fi
          }

Error in prepare-apisix container

---

Warning! Request etcd endpoint 'https://apisix-etcd.hostname:port/version' error, sslv3 alert bad certificate, retry time=60

---

This is fixed with adding etcd.tls parameters in the configmap. But then fails in apisix container.

    etcd:
        tls:
          cert: /bitnami/certs/tls.crt
          key: /bitnami/certs/tls.key

Error in apisix container

---

SSL_do_handshake() failed (SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:SSL alert number 42), context: ngx.timer

---

I was able to resolve this error in 2.1.2 chart version by adding etcd.mtls parameters. But it doesn't seem to work with the latest version.

  etcd:
      mtls:
          key_file: /bitnami/certs/{{ .Values.controlPlane.tls.certKeyFilename }}
          cert_file: /bitnami/certs/{{ .Values.controlPlane.tls.certFilename }}
          ca_file: /bitnami/certs/{{ .Values.controlPlane.tls.certCAFilename }}

[raji032]

Apisix reference document for etcd mtls connection - https://apisix.apache.org/docs/apisix/3.7/mtls/#etcd-with-mtls

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[joancafom]

Hi @raji032

This was time-consuming to debug, but I have partially reproduced your issue. I would like to know more about the SSL certificates you are using for APISIX and ETCD. Are they issued by the same CA? Are you using the auto-generated SSL certificates feature for APISIX and other completely different ones for ETCD?

It is totally fine if that is the case, I think we are missing some parameters to allow real mTLS ETCD encrypted connections in our chart.

[raji032]

Hi @joancafom, Thanks for the update. The CA for both apisix and etcd are same. The certs are generated using cert-manager and referenced the secrets as existing secrets in values.yaml.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.