使用外置etcd开启tls和认证后，启动报错

[SoHuDrgon]

我使用cfssl建立了证书：

echo '{"CN":"CA","key":{"algo":"rsa","size":2048}}' | cfssl gencert -initca - | cfssljson -bare ca -
echo '{"signing":{"default":{"expiry":"876000h","usages":["signing","key encipherment","server auth","client auth"]}}}' > ca-config.json
export ADDRESS=xxx-vapp-2012,172.16.20.12,127.0.0.1
export NAME=etcd-server
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME
export ADDRESS=xxx-app-38,xxx-app-39,xxx-app-50,xxx-vapp-2012,172.16.1.38,172.16.1.39,172.16.1.50,172.16.20.12,127.0.0.1
export NAME=etcd-client
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME

启动etcd后自测正常：

[root@xxx-app-38 apisix]# etcdctl --user=root         --password='xxx'        --cert /opt/apisix/ssl/etcd-client.pem         --key /opt/apisix/ssl/etcd-client-key.pem         --cacert /opt/apisix/ssl/ca.pem         --endpoints="https://172.16.20.12:2379"         auth status
Authentication Status: true
AuthRevision: 8

我在apisix的.values.yaml配置了：

# -- external etcd configuration. If etcd.enabled is false, these configuration will be used.
externalEtcd:
  # -- if etcd.enabled is false, use external etcd, support multiple address, if your etcd cluster enables TLS, please use https scheme, e.g. https://127.0.0.1:2379.
  host:
    # host or ip e.g. http://172.20.128.89:2379
    - https://172.16.20.12:2379
  # -- if etcd.enabled is false, user for external etcd. Set empty to disable authentication
  user: root
  # -- if etcd.enabled is true, use etcd.auth.rbac.rootPassword instead.
  # -- if etcd.enabled is false and externalEtcd.existingSecret is not empty, the password should store in the corresponding secret
  # -- if etcd.enabled is false and externalEtcd.existingSecret is empty, externalEtcd.password is the passsword for external etcd.
  password: ""
  # -- if externalEtcd.existingSecret is the name of secret containing the external etcd password
  existingSecret: "apisix-etcd-cert"
  # -- externalEtcd.secretPasswordKey Key inside the secret containing the external etcd password
  secretPasswordKey: "etcd-root-password"
  tls:
    enable: true
    verify: true
    caFilename: "ca.crt"
    certFilename: "tls.crt"
    certKeyFilename: "tls.key"
    sni: "xxx-vapp-2012"

并创建了apisix-etcd-cert：

kubectl -n apisix create secret generic apisix-etcd-cert \
  --from-literal=etcd-root-password='xxxxx' \
  --from-file=ca.crt=/opt/apisix/ssl//ca.pem \
  --from-file=tls.crt=/opt/apisix/ssl/etcd-client.pem \
  --from-file=tls.key=/opt/apisix/ssl/etcd-client-key.pem

安装apisix后报错：

# helm  --namespace apisix install apisix ./ -f values.yaml 
NAME: apisix
LAST DEPLOYED: Wed Oct 16 17:45:06 2024
NAMESPACE: apisix
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace apisix -o jsonpath="{.spec.ports[0].nodePort}" services apisix-gateway)
  export NODE_IP=$(kubectl get nodes --namespace apisix -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT
[root@xxx-app-38 apisix]# kubectl -n apisix logs apisix-6b5d9b65d8-czjfp 
/usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua init
/usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
request etcd endpoint 'https://172.16.20.12:2379/version' error, error loading private key (system lib)
all etcd nodes are unavailable
Warning! Request etcd endpoint 'https://172.16.20.12:2379/version' error, error loading private key ((null)), retry time=1
Warning! Request etcd endpoint 'https://172.16.20.12:2379/version' error, error loading private key (system lib), retry time=2

请问这是什么原因导致的，需要什么日志我都能提供！

[SoHuDrgon]

我使用这个方式创建secret：

kubectl create secret tls  etcd-tls --key etcd-client-key.pem --cert etcd-client.pem -n apisix

values.yaml配置如下：

# -- external etcd configuration. If etcd.enabled is false, these configuration will be used.
externalEtcd:
  # -- if etcd.enabled is false, use external etcd, support multiple address, if your etcd cluster enables TLS, please use https scheme, e.g. https://127.0.0.1:2379.
  host:
    # host or ip e.g. http://172.20.128.89:2379
    - https://172.16.20.12:2379
  # -- if etcd.enabled is false, user for external etcd. Set empty to disable authentication
  user: root
  # -- if etcd.enabled is true, use etcd.auth.rbac.rootPassword instead.
  # -- if etcd.enabled is false and externalEtcd.existingSecret is not empty, the password should store in the corresponding secret
  # -- if etcd.enabled is false and externalEtcd.existingSecret is empty, externalEtcd.password is the passsword for external etcd.
  password: ""
  # -- if externalEtcd.existingSecret is the name of secret containing the external etcd password
  existingSecret: "apisix-etcd-secret"
  # -- externalEtcd.secretPasswordKey Key inside the secret containing the external etcd password
  secretPasswordKey: "etcd-root-password"

# -- etcd configuration
# use the FQDN address or the IP of the etcd
etcd:
  # -- install etcd(v3) by default, set false if do not want to install etcd(v3) together
  enabled: false
  # -- apisix configurations prefix
  prefix: "/apisix"
  # -- Set the timeout value in seconds for subsequent socket operations from apisix to etcd cluster
  timeout: 30

  # -- if etcd.enabled is true, set more values of bitnami/etcd helm chart
  auth:
    rbac:
      # -- No authentication by default. Switch to enable RBAC authentication
      create: true
      # -- root password for etcd. Requires etcd.auth.rbac.create to be true.
      rootPassword: "xxxxxx"
    tls:
      # -- enable etcd client certificate
      enabled: true
      # -- name of the secret contains etcd client cert
      existingSecret: "etcd-tls"
      # -- etcd client cert filename using in etcd.auth.tls.existingSecret
      certFilename: "tls.crt"
      # -- etcd client cert key filename using in etcd.auth.tls.existingSecret
      certKeyFilename: "tls.key"
      # -- whether to verify the etcd endpoint certificate when setup a TLS connection to etcd
      verify: true
      # -- specify the TLS Server Name Indication extension, the ETCD endpoint hostname will be used when this setting is unset.
      sni: ""

安装apisix后：

# kubectl -n apisix logs apisix-c5967d8cd-n8h8h 
/usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua init
/usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
request etcd endpoint 'https://172.16.20.12:2379/version' error, certificate verify failed
all etcd nodes are unavailable
Warning! Request etcd endpoint 'https://172.16.20.12:2379/version' error, certificate verify failed, retry time=1
Warning! Request etcd endpoint 'https://172.16.20.12:2379/version' error, certificate verify failed, retry time=2

还是报错

[loresding]

需要在配置文件configmap.yaml的apisix.ssl.ssl_trust_certificate 执行ca.crt或者ca.pem，亲测有效