apisix icon indicating copy to clipboard operation
apisix copied to clipboard

Published 20 hours ago • verified publisher icon apache

bug: invalid number of arguments in "env" directive in /usr/local/apisix/conf/nginx.conf:29

Open BFergerson opened this issue 3 years ago • 25 comments

Current Behavior

I'm trying to dynamically set upstreams.tls.client_cert/client_key in stand-alone mode. I can't find a way to set the value via file lookup so I'm trying to set it via environment value. Whenever I set the value to the certificate value I get:

2022-06-09T07:06:59.109311603Z /usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init
2022-06-09T07:06:59.143984084Z /usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
2022-06-09T07:06:59.164148394Z 2022/06/09 07:06:59 [emerg] 1#1: invalid number of arguments in "env" directive in /usr/local/apisix/conf/nginx.conf:29
2022-06-09T07:06:59.164159537Z nginx: [emerg] invalid number of arguments in "env" directive in /usr/local/apisix/conf/nginx.conf:29

Expected Behavior

No response

Error Logs

No response

Steps to Reproduce

  1. Use config.yaml:
apisix:
  node_listen:
    - port: 80
  enable_admin: false
  config_center: yaml
  enable_ipv6: false
  1. Use apisix.yaml:
upstreams:
  -
    id: 1
    nodes:
      "httpbin.org:80": 1
    type: roundrobin
    scheme: grpcs
    tls:
      client_cert: ${{CCERT}}
      client_key: ${{CKEY}}
#END
  1. Use Dockerfile:
FROM apache/apisix:2.14.1-alpine

ENV CCERT="-----BEGIN CERTIFICATE-----\nMIIDOjCCAiICAwD6zzANBgkqhkiG9w0BAQsFADBnMQswCQYDVQQGEwJjbjESMBAG\nA1UECAwJR3VhbmdEb25nMQ8wDQYDVQQHDAZaaHVIYWkxDTALBgNVBAoMBGFwaTcx\nDDAKBgNVBAsMA29wczEWMBQGA1UEAwwNY2EuYXBpc2l4LmRldjAeFw0yMDA2MjAx\nMzE1MDBaFw0zMDA3MDgxMzE1MDBaMF0xCzAJBgNVBAYTAmNuMRIwEAYDVQQIDAlH\ndWFuZ0RvbmcxDTALBgNVBAoMBGFwaTcxDzANBgNVBAcMBlpodUhhaTEaMBgGA1UE\nAwwRY2xpZW50LmFwaXNpeC5kZXYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK\nAoIBAQCfKI8uiEH/ifZikSnRa3/E2B4ohVWRwjo/IxyDEWomgR4tLk1pSJhP/4SC\nLWuMQTFWTbSqt1IFYy4ZbVSHHyGoNPmJGrHRJCGE+sgpfzn0GjV4lXQPJD0k6GR1\nCX2Mo1TWdFqSJ/Hc5AQwcQFnPfoLAwsBy4yqrlmf96ZAUytl/7Zkjf4P7mJkJHtM\n/WgSR0pGhjZTAGRf5DJWoO51ki3i3JI+15mOhmnnCpnksnGVPfl92q92Hz/4v3iq\nE+UThPYRpcGbnddzMvPaCXiavg8B/u2LVbn4l0adamqQGepOAjD/1xraOVP2W22W\n0PztDXJ4rLe+capNS4oGuSUfkIENAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAHKn\nHxUhuk/nL2Sg5UB84OoJe5XPgNBvVMKN0c/NAPKVIPninvUcG/mHeKexPzE0sMga\nRNos75N2199EXydqUcsJ8jL0cNtQ2k5JQXXg0ntNC4tuCgIKAOnO879y5hSG36e5\n7wmAoVKnabgjej09zG1kkXvAmpgqoxeVCu7h7fK+AurLbsGCTaHoA5pG1tcHDxJQ\nfpVcbBfwQDSBW3SQjiRqX453/01nw6kbOeLKYraJysaG8ZU2K8+WpW6JDubciHjw\nfQnpU2U16XKivhxeuKYrV/INL0sxj/fZraNYErvJWzh5llvIdNLmeSPmvb50JUIs\n+lDqn1MobTXzDpuCFXA=\n-----END CERTIFICATE-----\n"
ENV CKEY="-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEAnyiPLohB/4n2YpEp0Wt/xNgeKIVVkcI6PyMcgxFqJoEeLS5N\naUiYT/+Egi1rjEExVk20qrdSBWMuGW1Uhx8hqDT5iRqx0SQhhPrIKX859Bo1eJV0\nDyQ9JOhkdQl9jKNU1nRakifx3OQEMHEBZz36CwMLAcuMqq5Zn/emQFMrZf+2ZI3+\nD+5iZCR7TP1oEkdKRoY2UwBkX+QyVqDudZIt4tySPteZjoZp5wqZ5LJxlT35fdqv\ndh8/+L94qhPlE4T2EaXBm53XczLz2gl4mr4PAf7ti1W5+JdGnWpqkBnqTgIw/9ca\n2jlT9lttltD87Q1yeKy3vnGqTUuKBrklH5CBDQIDAQABAoIBAHDe5bPdQ9jCcW3z\nfpGax/DER5b6//UvpfkSoGy/E+Wcmdb2yEVLC2FoVwOuzF+Z+DA5SU/sVAmoDZBQ\nvapZxJeygejeeo5ULkVNSFhNdr8LOzJ54uW+EHK1MFDj2xq61jaEK5sNIvRA7Eui\nSJl8FXBrxwmN3gNJRBwzF770fImHUfZt0YU3rWKw5Qin7QnlUzW2KPUltnSEq/xB\nkIzyWpuj7iAm9wTjH9Vy06sWCmxj1lzTTXlanjPb1jOTaOhbQMpyaAzRgQN8PZiE\nYKCarzVj7BJr7/vZYpnQtQDY12UL5n33BEqMP0VNHVqv+ZO3bktfvlwBru5ZJ7Cf\nURLsSc0CgYEAyz7FzV7cZYgjfUFD67MIS1HtVk7SX0UiYCsrGy8zA19tkhe3XVpc\nCZSwkjzjdEk0zEwiNAtawrDlR1m2kverbhhCHqXUOHwEpujMBjeJCNUVEh3OABr8\nvf2WJ6D1IRh8FA5CYLZP7aZ41fcxAnvIPAEThemLQL3C4H5H5NG2WFsCgYEAyHhP\nonpS/Eo/OXKYFLR/mvjizRVSomz1lVVL+GWMUYQsmgsPyBJgyAOX3Pqt9catgxhM\nDbEr7EWTxth3YeVzamiJPNVK0HvCax9gQ0KkOmtbrfN54zBHOJ+ieYhsieZLMgjx\niu7Ieo6LDGV39HkvekzutZpypiCpKlMaFlCFiLcCgYEAmAgRsEj4Nh665VPvuZzH\nZIgZMAlwBgHR7/v6l7AbybcVYEXLTNJtrGEEH6/aOL8V9ogwwZuIvb/TEidCkfcf\nzg/pTcGf2My0MiJLk47xO6EgzNdso9mMG5ZYPraBBsuo7NupvWxCp7NyCiOJDqGH\nK5NmhjInjzsjTghIQRq5+qcCgYEAxnm/NjjvslL8F69p/I3cDJ2/RpaG0sMXvbrO\nVWaMryQyWGz9OfNgGIbeMu2Jj90dar6ChcfUmb8lGOi2AZl/VGmc/jqaMKFnElHl\nJ5JyMFicUzPMiG8DBH+gB71W4Iy+BBKwugHBQP2hkytewQ++PtKuP+RjADEz6vCN\n0mv0WS8CgYBnbMRP8wIOLJPRMw/iL9BdMf606X4xbmNn9HWVp2mH9D3D51kDFvls\n7y2vEaYkFv3XoYgVN9ZHDUbM/YTUozKjcAcvz0syLQb8wRwKeo+XSmo09+360r18\nzRugoE7bPl39WdGWaW3td0qf1r9z3sE2iWUTJPRQ3DYpsLOYIgyKmw==\n-----END RSA PRIVATE KEY-----\n"

ADD ./config.yaml /usr/local/apisix/conf/config.yaml
ADD ./apisix.yaml /usr/local/apisix/conf/apisix.yaml

CCERT and CKEY are taken from https://github.com/apache/apisix/blob/1b0c182ea007acccaabda3c13d7f4102da3944d9/t/node/grpc-proxy-mtls.t#L88-L89

Environment

  • APISIX version (run apisix version): apache/apisix:2.14.1-alpine

BFergerson avatar Jun 09 '22 07:06 BFergerson

@spacewander @tzssangglass Please take a look when you have time.

tokers avatar Jun 09 '22 09:06 tokers

I reproduced it, and the problem was with this paragraph in nginx.conf:

env APISIX_PROFILE;
env PATH; # for searching external plugin runner's binary

env CKEY=-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEAnyiPLohB/4n2YpEp0Wt/xNgeKIVVkcI6PyMcgxFqJoEeLS5N\naUiYT/+Egi1rjEExVk20qrdSBWMuGW1Uhx8hqDT5iRqx0SQhhPrIKX859Bo1eJV0\nDyQ9JOhkdQl9jKNU1nRakifx3OQEMHEBZz36CwMLAcuMqq5Zn/emQFMrZf+2ZI3+\nD+5iZCR7TP1oEkdKRoY2UwBkX+QyVqDudZIt4tySPteZjoZp5wqZ5LJxlT35fdqv\ndh8/+L94qhPlE4T2EaXBm53XczLz2gl4mr4PAf7ti1W5+JdGnWpqkBnqTgIw/9ca\n2jlT9lttltD87Q1yeKy3vnGqTUuKBrklH5CBDQIDAQABAoIBAHDe5bPdQ9jCcW3z\nfpGax/DER5b6//UvpfkSoGy/E+Wcmdb2yEVLC2FoVwOuzF+Z+DA5SU/sVAmoDZBQ\nvapZxJeygejeeo5ULkVNSFhNdr8LOzJ54uW+EHK1MFDj2xq61jaEK5sNIvRA7Eui\nSJl8FXBrxwmN3gNJRBwzF770fImHUfZt0YU3rWKw5Qin7QnlUzW2KPUltnSEq/xB\nkIzyWpuj7iAm9wTjH9Vy06sWCmxj1lzTTXlanjPb1jOTaOhbQMpyaAzRgQN8PZiE\nYKCarzVj7BJr7/vZYpnQtQDY12UL5n33BEqMP0VNHVqv+ZO3bktfvlwBru5ZJ7Cf\nURLsSc0CgYEAyz7FzV7cZYgjfUFD67MIS1HtVk7SX0UiYCsrGy8zA19tkhe3XVpc\nCZSwkjzjdEk0zEwiNAtawrDlR1m2kverbhhCHqXUOHwEpujMBjeJCNUVEh3OABr8\nvf2WJ6D1IRh8FA5CYLZP7aZ41fcxAnvIPAEThemLQL3C4H5H5NG2WFsCgYEAyHhP\nonpS/Eo/OXKYFLR/mvjizRVSomz1lVVL+GWMUYQsmgsPyBJgyAOX3Pqt9catgxhM\nDbEr7EWTxth3YeVzamiJPNVK0HvCax9gQ0KkOmtbrfN54zBHOJ+ieYhsieZLMgjx\niu7Ieo6LDGV39HkvekzutZpypiCpKlMaFlCFiLcCgYEAmAgRsEj4Nh665VPvuZzH\nZIgZMAlwBgHR7/v6l7AbybcVYEXLTNJtrGEEH6/aOL8V9ogwwZuIvb/TEidCkfcf\nzg/pTcGf2My0MiJLk47xO6EgzNdso9mMG5ZYPraBBsuo7NupvWxCp7NyCiOJDqGH\nK5NmhjInjzsjTghIQRq5+qcCgYEAxnm/NjjvslL8F69p/I3cDJ2/RpaG0sMXvbrO\nVWaMryQyWGz9OfNgGIbeMu2Jj90dar6ChcfUmb8lGOi2AZl/VGmc/jqaMKFnElHl\nJ5JyMFicUzPMiG8DBH+gB71W4Iy+BBKwugHBQP2hkytewQ++PtKuP+RjADEz6vCN\n0mv0WS8CgYBnbMRP8wIOLJPRMw/iL9BdMf606X4xbmNn9HWVp2mH9D3D51kDFvls\n7y2vEaYkFv3XoYgVN9ZHDUbM/YTUozKjcAcvz0syLQb8wRwKeo+XSmo09+360r18\nzRugoE7bPl39WdGWaW3td0qf1r9z3sE2iWUTJPRQ3DYpsLOYIgyKmw==\n-----END RSA PRIVATE KEY-----\n;
env CCERT=-----BEGIN CERTIFICATE-----\nMIIDOjCCAiICAwD6zzANBgkqhkiG9w0BAQsFADBnMQswCQYDVQQGEwJjbjESMBAG\nA1UECAwJR3VhbmdEb25nMQ8wDQYDVQQHDAZaaHVIYWkxDTALBgNVBAoMBGFwaTcx\nDDAKBgNVBAsMA29wczEWMBQGA1UEAwwNY2EuYXBpc2l4LmRldjAeFw0yMDA2MjAx\nMzE1MDBaFw0zMDA3MDgxMzE1MDBaMF0xCzAJBgNVBAYTAmNuMRIwEAYDVQQIDAlH\ndWFuZ0RvbmcxDTALBgNVBAoMBGFwaTcxDzANBgNVBAcMBlpodUhhaTEaMBgGA1UE\nAwwRY2xpZW50LmFwaXNpeC5kZXYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK\nAoIBAQCfKI8uiEH/ifZikSnRa3/E2B4ohVWRwjo/IxyDEWomgR4tLk1pSJhP/4SC\nLWuMQTFWTbSqt1IFYy4ZbVSHHyGoNPmJGrHRJCGE+sgpfzn0GjV4lXQPJD0k6GR1\nCX2Mo1TWdFqSJ/Hc5AQwcQFnPfoLAwsBy4yqrlmf96ZAUytl/7Zkjf4P7mJkJHtM\n/WgSR0pGhjZTAGRf5DJWoO51ki3i3JI+15mOhmnnCpnksnGVPfl92q92Hz/4v3iq\nE+UThPYRpcGbnddzMvPaCXiavg8B/u2LVbn4l0adamqQGepOAjD/1xraOVP2W22W\n0PztDXJ4rLe+capNS4oGuSUfkIENAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAHKn\nHxUhuk/nL2Sg5UB84OoJe5XPgNBvVMKN0c/NAPKVIPninvUcG/mHeKexPzE0sMga\nRNos75N2199EXydqUcsJ8jL0cNtQ2k5JQXXg0ntNC4tuCgIKAOnO879y5hSG36e5\n7wmAoVKnabgjej09zG1kkXvAmpgqoxeVCu7h7fK+AurLbsGCTaHoA5pG1tcHDxJQ\nfpVcbBfwQDSBW3SQjiRqX453/01nw6kbOeLKYraJysaG8ZU2K8+WpW6JDubciHjw\nfQnpU2U16XKivhxeuKYrV/INL0sxj/fZraNYErvJWzh5llvIdNLmeSPmvb50JUIs\n+lDqn1MobTXzDpuCFXA=\n-----END CERTIFICATE-----\n;


http {
    # put extra_lua_path in front of the builtin path
……

I will check it.

tzssangglass avatar Jun 09 '22 12:06 tzssangglass

APISIX startup phase gets the environment variables(CCERT , CCERT ) in apisix.yaml here: https://github.com/apache/apisix/blob/975038ea70dc47e0613c42b3eabb8dcc730216a4/apisix/cli/ops.lua#L646

and the environment variable is used here when generating nginx.conf: https://github.com/apache/apisix/blob/975038ea70dc47e0613c42b3eabb8dcc730216a4/apisix/cli/ngx_tpl.lua#L54-L58

tzssangglass avatar Jun 09 '22 12:06 tzssangglass

APISIX startup phase gets the environment variables(CCERT , CCERT ) in apisix.yaml here:

https://github.com/apache/apisix/blob/975038ea70dc47e0613c42b3eabb8dcc730216a4/apisix/cli/ops.lua#L646

and the environment variable is used here when generating nginx.conf:

https://github.com/apache/apisix/blob/975038ea70dc47e0613c42b3eabb8dcc730216a4/apisix/cli/ngx_tpl.lua#L54-L58

Looks like the name is empty so the env directive is incomplete.

tokers avatar Jun 10 '22 01:06 tokers

First I did some verification that passing a normal string variable with env works.

apisix.yaml

upstreams:
  -
    id: 1
    nodes:
      "httpbin.org": 1
    type: roundrobin
    scheme: ${{UP_SCHEME}}
routes:
  -
    uri: /*
    upstream_id: 1
#END

config.yaml

apisix:
  node_listen:
    - port: 80
  enable_admin: false
  config_center: yaml
  enable_ipv6: false

test 1 set env like

export UP_SCHEME=http

UP_SCHEME in nginx.conf like

env UP_SCHEME=http;

curl http://127.0.0.1:80/get -i
HTTP/1.1 200 OK

test 2 set env like

export UP_SCHEME=grpc

UP_SCHEME in nginx.conf like

env UP_SCHEME=grpc;

curl http://127.0.0.1:80/get -i
HTTP/1.1 502 Bad Gateway

Passing the certificate through the environment variable failed, it should be because you passed the certificate with a newline character \n.

tzssangglass avatar Jun 10 '22 05:06 tzssangglass

@tzssangglass, thanks for looking into this but are you sure it's the \n? My original code does not contain the \n. I only added them after I found the certificates embedded in APISIX using them.

Here are certificates without \n that cause the same issue:

ENV CCERT="-----BEGIN CERTIFICATE-----MIIDOjCCAiICAwD6zzANBgkqhkiG9w0BAQsFADBnMQswCQYDVQQGEwJjbjESMBAGA1UECAwJR3VhbmdEb25nMQ8wDQYDVQQHDAZaaHVIYWkxDTALBgNVBAoMBGFwaTcxDDAKBgNVBAsMA29wczEWMBQGA1UEAwwNY2EuYXBpc2l4LmRldjAeFw0yMDA2MjAxMzE1MDBaFw0zMDA3MDgxMzE1MDBaMF0xCzAJBgNVBAYTAmNuMRIwEAYDVQQIDAlHdWFuZ0RvbmcxDTALBgNVBAoMBGFwaTcxDzANBgNVBAcMBlpodUhhaTEaMBgGA1UEAwwRY2xpZW50LmFwaXNpeC5kZXYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCfKI8uiEH/ifZikSnRa3/E2B4ohVWRwjo/IxyDEWomgR4tLk1pSJhP/4SCLWuMQTFWTbSqt1IFYy4ZbVSHHyGoNPmJGrHRJCGE+sgpfzn0GjV4lXQPJD0k6GR1CX2Mo1TWdFqSJ/Hc5AQwcQFnPfoLAwsBy4yqrlmf96ZAUytl/7Zkjf4P7mJkJHtM/WgSR0pGhjZTAGRf5DJWoO51ki3i3JI+15mOhmnnCpnksnGVPfl92q92Hz/4v3iqE+UThPYRpcGbnddzMvPaCXiavg8B/u2LVbn4l0adamqQGepOAjD/1xraOVP2W22W0PztDXJ4rLe+capNS4oGuSUfkIENAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAHKnHxUhuk/nL2Sg5UB84OoJe5XPgNBvVMKN0c/NAPKVIPninvUcG/mHeKexPzE0sMgaRNos75N2199EXydqUcsJ8jL0cNtQ2k5JQXXg0ntNC4tuCgIKAOnO879y5hSG36e57wmAoVKnabgjej09zG1kkXvAmpgqoxeVCu7h7fK+AurLbsGCTaHoA5pG1tcHDxJQfpVcbBfwQDSBW3SQjiRqX453/01nw6kbOeLKYraJysaG8ZU2K8+WpW6JDubciHjwfQnpU2U16XKivhxeuKYrV/INL0sxj/fZraNYErvJWzh5llvIdNLmeSPmvb50JUIs+lDqn1MobTXzDpuCFXA=-----END CERTIFICATE-----"
ENV CKEY="-----BEGIN RSA PRIVATE KEY-----MIIEpAIBAAKCAQEAnyiPLohB/4n2YpEp0Wt/xNgeKIVVkcI6PyMcgxFqJoEeLS5NaUiYT/+Egi1rjEExVk20qrdSBWMuGW1Uhx8hqDT5iRqx0SQhhPrIKX859Bo1eJV0DyQ9JOhkdQl9jKNU1nRakifx3OQEMHEBZz36CwMLAcuMqq5Zn/emQFMrZf+2ZI3+D+5iZCR7TP1oEkdKRoY2UwBkX+QyVqDudZIt4tySPteZjoZp5wqZ5LJxlT35fdqvdh8/+L94qhPlE4T2EaXBm53XczLz2gl4mr4PAf7ti1W5+JdGnWpqkBnqTgIw/9ca2jlT9lttltD87Q1yeKy3vnGqTUuKBrklH5CBDQIDAQABAoIBAHDe5bPdQ9jCcW3zfpGax/DER5b6//UvpfkSoGy/E+Wcmdb2yEVLC2FoVwOuzF+Z+DA5SU/sVAmoDZBQvapZxJeygejeeo5ULkVNSFhNdr8LOzJ54uW+EHK1MFDj2xq61jaEK5sNIvRA7EuiSJl8FXBrxwmN3gNJRBwzF770fImHUfZt0YU3rWKw5Qin7QnlUzW2KPUltnSEq/xBkIzyWpuj7iAm9wTjH9Vy06sWCmxj1lzTTXlanjPb1jOTaOhbQMpyaAzRgQN8PZiEYKCarzVj7BJr7/vZYpnQtQDY12UL5n33BEqMP0VNHVqv+ZO3bktfvlwBru5ZJ7CfURLsSc0CgYEAyz7FzV7cZYgjfUFD67MIS1HtVk7SX0UiYCsrGy8zA19tkhe3XVpcCZSwkjzjdEk0zEwiNAtawrDlR1m2kverbhhCHqXUOHwEpujMBjeJCNUVEh3OABr8vf2WJ6D1IRh8FA5CYLZP7aZ41fcxAnvIPAEThemLQL3C4H5H5NG2WFsCgYEAyHhPonpS/Eo/OXKYFLR/mvjizRVSomz1lVVL+GWMUYQsmgsPyBJgyAOX3Pqt9catgxhMDbEr7EWTxth3YeVzamiJPNVK0HvCax9gQ0KkOmtbrfN54zBHOJ+ieYhsieZLMgjxiu7Ieo6LDGV39HkvekzutZpypiCpKlMaFlCFiLcCgYEAmAgRsEj4Nh665VPvuZzHZIgZMAlwBgHR7/v6l7AbybcVYEXLTNJtrGEEH6/aOL8V9ogwwZuIvb/TEidCkfcfzg/pTcGf2My0MiJLk47xO6EgzNdso9mMG5ZYPraBBsuo7NupvWxCp7NyCiOJDqGHK5NmhjInjzsjTghIQRq5+qcCgYEAxnm/NjjvslL8F69p/I3cDJ2/RpaG0sMXvbrOVWaMryQyWGz9OfNgGIbeMu2Jj90dar6ChcfUmb8lGOi2AZl/VGmc/jqaMKFnElHlJ5JyMFicUzPMiG8DBH+gB71W4Iy+BBKwugHBQP2hkytewQ++PtKuP+RjADEz6vCN0mv0WS8CgYBnbMRP8wIOLJPRMw/iL9BdMf606X4xbmNn9HWVp2mH9D3D51kDFvls7y2vEaYkFv3XoYgVN9ZHDUbM/YTUozKjcAcvz0syLQb8wRwKeo+XSmo09+360r18zRugoE7bPl39WdGWaW3td0qf1r9z3sE2iWUTJPRQ3DYpsLOYIgyKmw==-----END RSA PRIVATE KEY-----"

BFergerson avatar Jun 10 '22 05:06 BFergerson

Looks like it's actually the -----BEGIN CERTIFICATE----- part.

BFergerson avatar Jun 10 '22 05:06 BFergerson

Passing the certificate through the environment variable failed, it should be because you passed the certificate with a newline character \n.

To verify it, I did the following tests:

set env var like

export CCERT="-----BEGIN CERTIFICATE-----\nMIIEojCCAwqgAwIBAgIJAK253pMhgCkxMA0GCSqGSIb3DQEBCwUAMFYxCzAJBgNV\nBAYTAkNOMRIwEAYDVQQIDAlHdWFuZ0RvbmcxDzANBgNVBAcMBlpodUhhaTEPMA0G\nA1UECgwGaXJlc3R5MREwDwYDVQQDDAh0ZXN0LmNvbTAgFw0xOTA2MjQyMjE4MDVa\nGA8yMTE5MDUzMTIyMTgwNVowVjELMAkGA1UEBhMCQ04xEjAQBgNVBAgMCUd1YW5n\nRG9uZzEPMA0GA1UEBwwGWmh1SGFpMQ8wDQYDVQQKDAZpcmVzdHkxETAPBgNVBAMM\nCHRlc3QuY29tMIIBojANBgkqhkiG9w0BAQEFAAOCAY8AMIIBigKCAYEAyCM0rqJe\ncvgnCfOw4fATotPwk5Ba0gC2YvIrO+gSbQkyxXF5jhZB3W6BkWUWR4oNFLLSqcVb\nVDPitz/Mt46Mo8amuS6zTbQetGnBARzPLtmVhJfoeLj0efMiOepOSZflj9Ob4yKR\n2bGdEFOdHPjm+4ggXU9jMKeLqdVvxll/JiVFBW5smPtW1Oc/BV5terhscJdOgmRr\nabf9xiIis9/qVYfyGn52u9452V0owUuwP7nZ01jt6iMWEGeQU6mwPENgvj1olji2\nWjdG2UwpUVp3jp3l7j1ekQ6mI0F7yI+LeHzfUwiyVt1TmtMWn1ztk6FfLRqwJWR/\nEvm95vnfS3Le4S2ky3XAgn2UnCMyej3wDN6qHR1onpRVeXhrBajbCRDRBMwaNw/1\n/3Uvza8QKK10PzQR6OcQ0xo9psMkd9j9ts/dTuo2fzaqpIfyUbPST4GdqNG9NyIh\n/B9g26/0EWcjyO7mYVkaycrtLMaXm1u9jyRmcQQI1cGrGwyXbrieNp63AgMBAAGj\ncTBvMB0GA1UdDgQWBBSZtSvV8mBwl0bpkvFtgyiOUUcbszAfBgNVHSMEGDAWgBSZ\ntSvV8mBwl0bpkvFtgyiOUUcbszAMBgNVHRMEBTADAQH/MB8GA1UdEQQYMBaCCHRl\nc3QuY29tggoqLnRlc3QuY29tMA0GCSqGSIb3DQEBCwUAA4IBgQAHGEul/x7ViVgC\ntC8CbXEslYEkj1XVr2Y4hXZXAXKd3W7V3TC8rqWWBbr6L/tsSVFt126V5WyRmOaY\n1A5pju8VhnkhYxYfZALQxJN2tZPFVeME9iGJ9BE1wPtpMgITX8Rt9kbNlENfAgOl\nPYzrUZN1YUQjX+X8t8/1VkSmyZysr6ngJ46/M8F16gfYXc9zFj846Z9VST0zCKob\nrJs3GtHOkS9zGGldqKKCj+Awl0jvTstI4qtS1ED92tcnJh5j/SSXCAB5FgnpKZWy\nhme45nBQj86rJ8FhN+/aQ9H9/2Ib6Q4wbpaIvf4lQdLUEcWAeZGW6Rk0JURwEog1\n7/mMgkapDglgeFx9f/XztSTrkHTaX4Obr+nYrZ2V4KOB4llZnK5GeNjDrOOJDk2y\nIJFgBOZJWyS93dQfuKEj42hA79MuX64lMSCVQSjX+ipR289GQZqFrIhiJxLyA+Ve\nU/OOcSRr39Kuis/JJ+DkgHYa/PWHZhnJQBxcqXXk1bJGw9BNbhM=\n-----END CERTIFICATE-----\n"
export CKEY="HrMHUvE9Esvn7GnZ+vAynaIg/8wlB3r0zm0htmnwofYLp1VhtLeU1EmMJkPLUkcn2+v6Uav9bOQMkPdSpUMcEpRplLSXs+miu+B07CCUnsMrXkfQawRMIoePJZSLH5+PfDAlWIK2Q+ruYnjtnpNziiAtXf/HRRwHHMelnfedXqD8kn3Toe46ZYyBir99o/r/do5ludez5oY7qhOgNSWKCfnZE8Ip82g7t7n7jsAf5tTdRulUGBQ4ITV2zM3cxpD0PWnWMbOfygZIDxR8QU9wj8ihuFL1s1NM8PplcKbUxC4QlrSN+ZNkr6mxy+akPmXlABwcFIiSK7c/xvU1NjoILnhPpL6aRpbhmQX/a1XUCl+2INlQ5QbXbTN+JmDBhrU9NiYecRJMfmA1N/lhwgt01tUnxMoAhfpUVgEbZNalCJt+wn8TC+Xp3DZ0bCpXrfzqsprGKan9qC3mCN03jj50JyGFL+xt8wX8D0uaIsu4cVk4et7kbTIj9rvucsh0cfKn8va8/cdjw5QhFSRBkW5Vuz9NwvzVQ6DHWs1a8VZbN/hERxcbWNk/p1VgGLHioqZZTOd5CYdN4dGjnksjXa0Z77mTSoNx3U79FQPAgUMEA1phnO/jdryM3g5M+UvESXA/75we435xg5tLRDvNwJw2NlosQsGY7fzUi2+HFo436htydRFv8ChHezs2v99mjfCUijrWYoeJ5OB2+KO9XiOIz7gpqhTef9atajSYRhxhcwdCVupC1PrPGn9MzhdQLeqQCJj3kyazPfO3xPkNpMAqd2lXnLR4HGd9SBHe75Sik3jW9W1sUqrn2fDjyWd0jz57pl4qyHjbzjd3uE5qbH/QuYZBIzI9tEn7tj12brWrwHsMt+/4M7zp8Opsia64V3Y7ICLIi7fiYfr70RujXyn8Ik5TB1QC98JrnDjgQlTPDhHLk1r8XhZXqIIg6DmaN7UUjIuZhKxARTs8b5WMPvVV4GownlPN28sHIMAX84BNbP0597Fxipwp2oTMFKTzvxm+QUtbWvIPzF3n25L4sPCyUx5PRIRCJ5kDNQfhiN6o3Y/fAY0PyxI06PWYoNvSn3uO24XNXbF3RkpwKtV8n/iNo5dyM1VqFPWDuKRSLHY7E4lQTdqx4/n+rrnoH6SlmQ0zwxwxBeAz/TvkmiW7WLe3C5cUDKF9yYwvAe8ek4oTR3GxaiDWjNFsu7DUoDjpH5f3IxrX2IN4FyzE47hMeg4muPov7h74WwosqgnfmwoAEFV4+ldmzpdSjghZoF2M9EZI24Xa9rVdd6j2t6IjX20oL+SLQL/9HppMi1nC+3Zby1WOvuTR4g8K1QP75OeY4xTD1iEAXpd0WOX7C3ndceVF4THLCI4Imcf9FH9MBrE55FPMEsAk54HiAoyMd6tgqv/akRqmuAmnSsrWALhqiCnAVh2uzk644gSzmsFbh7zF33qrcafPpU4PxUEvpqbLz7asoNUDf4YB4gCcgZx30eK/w9FpMaLveiNq77EW7qcvJQPcjZ4uLaKkQVODJsd+1CbZF6370aiLxouXLFT3eQI7Ovu6be8D3MmazRPgCV36qzMwONqrXE/JbMFMKe5l1e4Y6avMejrj43BMgGo2u8LimCWkBeNwqIjH7plwbpDKo4OKZVbrzSZ0hplUDd/jMrb6Ulbc04uMeEigehrhSsZ0ZwoDiZcf/fDIclaTGNMl40N2wBiqdnw9uKTqD1YxzqDQ7vgiXG55ae31lvevPTgk/lLvpwzlyitjGs+6LJPu/wSCKA2VIyhJfK+8EnItEKjBUrXdOklBdOmTpUpdQ+zfd2NCrFRDJZKl26Uh412adFEkqY37O/0FbSCpAIsUCvaItcqK7qh5Rq26hVR0nS1MRs+MjGBzGqudXPQZHy+Yp7AlAa5UgJUaAwn2b/id6kNdv6hNWqSzHvOAVKdgC9/j0yN1VJD92+IoJTTiXsMQELcgm1Ehj2GZpTHu+GPuaOovHBnZMq/Kg4nUS+ig86X01jV28uGGtglERf1HqVQpdZwbrXtUqH0cbjlvUwQ1j7zp9yhs+0ta87v0I+elAZhXzqvehMiLJu2o9/k2+4dPvkEscduHOU6jZqe8ndNEMQWiaZEYJKxNWPTaQ6nZSlFTsT7GlENeJlFzlw8QkyRJPMBWkXuaymQUcu43Pm+gAjinHSAGUeaSaIdL2Yb0M88qNwG+UlNEslx/J37pA1oMJyxb7XOeySxkP7dXi5JvygLIfkEA3ENC4NHU9nsUvTvp5AZidZCxxtYCNYfjY6xyrlfnE+V+us31LA9Wc/tKa4y3Ldj30IT2sssUrdZ0l7UbwfcZT42ZeJpxDofpZ2rjgswTs0Upr72VuOCzjpKa1CJwxhVVtPVJJovcXp4bsNPJers+yIYfTl1aqaf4qSzU5OL/cze2e6qAh7622zEa/q6klpUx9b1f8YGlQhjQcy3++JnwwsHR71Ofh9woXq57LDCHFA6f95zdkadDDhwgRcvWVnbA2Szps8iJv7h2m25qZPFtN6puJj3RlmT6hnfBeYCjpfy/2TxyCqm6bG3HZxGuhzWs2ZGxzsjBJ3ueO1pAOjtDhkRqzoWt/v2o367IYP7iTcp4pi+qJHIWCN1ElDI0BVoZ+Xq9iLfKmjrjcxQ7EYGHfQDE52QaCQ3nMB7oiqncZ1Q5n/ICDHha9RkPP9V9vWiJIZwgOJtPfGzsGQ9AigH6po65IJyxmY5upuhg7DTmsLQnKC/fwjkBF9So/4cdZuqDbxGrDDOgpL7uvWXANRNMrqYoMFUG7M90QJHj7NgSL+B6mSNwa9ctTua7Estkoyvavda3Bl3qHQ0Hva5gjSg6elL6PQ4ksqhESvjztuy58qk9aZHsQB8ZKRu8VSay40a/3ueX6bnd0hwsYy42aWJR1z+uie3yTWPuG2JZ7DjkgDduWdC+cxfvTVTG58E5luafy5j/t85UVoB2nr46VHlt/vg4M9G8/4F0d0Y6ThI4/XTfg6l1vq5ouzhQxd+SRwnuXieZy+4/2XKJnrV6t+JbNAvwdGR1V9VPLlnb+IqpvOCYyL1YLYSlNubb9HU0wxVPppGSpJLmi+njQzl71PBgMm6QV9j889wPUo387fRbJjXbSSVLon61xk/4dNvjsgfv9rF+/qEML0q4tXBJVOJ1iwKjn84Nk6vdHM3Hu8knp0hYFa4AECYKInSTVXajWAKFx4SOq8G8MA/0YlIN872LBjUm2GKs17wsJuWID+mSyVE5pV5gQ+r92YvPcC+yIvB8hTTaRclAP/KyJesDTA=="

this cert and key from: https://github.com/apache/apisix/blob/master/t/cli/test_upstream_mtls.sh#L57-L74

it is the right configuration, and it is important to note that

  1. CCERT has line break
  2. CKEY has no line break

config.yaml

apisix:
  node_listen:
    - port: 80
  enable_admin: false
  config_center: yaml
  enable_ipv6: false

test 1

apisix.yaml only use CKEY

upstreams:
  -
    id: 1
    nodes:
      "httpbin.org": 1
    type: roundrobin
    scheme: http
    tls:
      client_key: ${{CKEY}}
routes:
  -
    uri: /*
    upstream_id: 1
#END

Can be started normally, no errors are reported in the configuration.

test 2

apisix.yaml use CKEY and CCERT

upstreams:
  -
    id: 1
    nodes:
      "httpbin.org": 1
    type: roundrobin
    scheme: http
    tls:
      client_cert: ${{CCERT}}
      client_key: ${{CKEY}}
routes:
  -
    uri: /*
    upstream_id: 1
#END

invalid number of arguments in "env" directive

This can prove to be related to line breaks.

tzssangglass avatar Jun 10 '22 05:06 tzssangglass

I only added them after I found the certificates embedded in APISIX using them.

Here are certificates without \n that cause the same issue:

I will try to find the available configuration forms.

tzssangglass avatar Jun 10 '22 05:06 tzssangglass

@tzssangglass, the following gets pass the error I reported:

ENV CCERT="\nMIIDOjCCAiICAwD6zzANBgkqhkiG9w0BAQsFADBnMQswCQYDVQQGEwJjbjESMBAG\nA1UECAwJR3VhbmdEb25nMQ8wDQYDVQQHDAZaaHVIYWkxDTALBgNVBAoMBGFwaTcx\nDDAKBgNVBAsMA29wczEWMBQGA1UEAwwNY2EuYXBpc2l4LmRldjAeFw0yMDA2MjAx\nMzE1MDBaFw0zMDA3MDgxMzE1MDBaMF0xCzAJBgNVBAYTAmNuMRIwEAYDVQQIDAlH\ndWFuZ0RvbmcxDTALBgNVBAoMBGFwaTcxDzANBgNVBAcMBlpodUhhaTEaMBgGA1UE\nAwwRY2xpZW50LmFwaXNpeC5kZXYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK\nAoIBAQCfKI8uiEH/ifZikSnRa3/E2B4ohVWRwjo/IxyDEWomgR4tLk1pSJhP/4SC\nLWuMQTFWTbSqt1IFYy4ZbVSHHyGoNPmJGrHRJCGE+sgpfzn0GjV4lXQPJD0k6GR1\nCX2Mo1TWdFqSJ/Hc5AQwcQFnPfoLAwsBy4yqrlmf96ZAUytl/7Zkjf4P7mJkJHtM\n/WgSR0pGhjZTAGRf5DJWoO51ki3i3JI+15mOhmnnCpnksnGVPfl92q92Hz/4v3iq\nE+UThPYRpcGbnddzMvPaCXiavg8B/u2LVbn4l0adamqQGepOAjD/1xraOVP2W22W\n0PztDXJ4rLe+capNS4oGuSUfkIENAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAHKn\nHxUhuk/nL2Sg5UB84OoJe5XPgNBvVMKN0c/NAPKVIPninvUcG/mHeKexPzE0sMga\nRNos75N2199EXydqUcsJ8jL0cNtQ2k5JQXXg0ntNC4tuCgIKAOnO879y5hSG36e5\n7wmAoVKnabgjej09zG1kkXvAmpgqoxeVCu7h7fK+AurLbsGCTaHoA5pG1tcHDxJQ\nfpVcbBfwQDSBW3SQjiRqX453/01nw6kbOeLKYraJysaG8ZU2K8+WpW6JDubciHjw\nfQnpU2U16XKivhxeuKYrV/INL0sxj/fZraNYErvJWzh5llvIdNLmeSPmvb50JUIs\n+lDqn1MobTXzDpuCFXA=\n\n"
ENV CKEY="\nMIIEpAIBAAKCAQEAnyiPLohB/4n2YpEp0Wt/xNgeKIVVkcI6PyMcgxFqJoEeLS5N\naUiYT/+Egi1rjEExVk20qrdSBWMuGW1Uhx8hqDT5iRqx0SQhhPrIKX859Bo1eJV0\nDyQ9JOhkdQl9jKNU1nRakifx3OQEMHEBZz36CwMLAcuMqq5Zn/emQFMrZf+2ZI3+\nD+5iZCR7TP1oEkdKRoY2UwBkX+QyVqDudZIt4tySPteZjoZp5wqZ5LJxlT35fdqv\ndh8/+L94qhPlE4T2EaXBm53XczLz2gl4mr4PAf7ti1W5+JdGnWpqkBnqTgIw/9ca\n2jlT9lttltD87Q1yeKy3vnGqTUuKBrklH5CBDQIDAQABAoIBAHDe5bPdQ9jCcW3z\nfpGax/DER5b6//UvpfkSoGy/E+Wcmdb2yEVLC2FoVwOuzF+Z+DA5SU/sVAmoDZBQ\nvapZxJeygejeeo5ULkVNSFhNdr8LOzJ54uW+EHK1MFDj2xq61jaEK5sNIvRA7Eui\nSJl8FXBrxwmN3gNJRBwzF770fImHUfZt0YU3rWKw5Qin7QnlUzW2KPUltnSEq/xB\nkIzyWpuj7iAm9wTjH9Vy06sWCmxj1lzTTXlanjPb1jOTaOhbQMpyaAzRgQN8PZiE\nYKCarzVj7BJr7/vZYpnQtQDY12UL5n33BEqMP0VNHVqv+ZO3bktfvlwBru5ZJ7Cf\nURLsSc0CgYEAyz7FzV7cZYgjfUFD67MIS1HtVk7SX0UiYCsrGy8zA19tkhe3XVpc\nCZSwkjzjdEk0zEwiNAtawrDlR1m2kverbhhCHqXUOHwEpujMBjeJCNUVEh3OABr8\nvf2WJ6D1IRh8FA5CYLZP7aZ41fcxAnvIPAEThemLQL3C4H5H5NG2WFsCgYEAyHhP\nonpS/Eo/OXKYFLR/mvjizRVSomz1lVVL+GWMUYQsmgsPyBJgyAOX3Pqt9catgxhM\nDbEr7EWTxth3YeVzamiJPNVK0HvCax9gQ0KkOmtbrfN54zBHOJ+ieYhsieZLMgjx\niu7Ieo6LDGV39HkvekzutZpypiCpKlMaFlCFiLcCgYEAmAgRsEj4Nh665VPvuZzH\nZIgZMAlwBgHR7/v6l7AbybcVYEXLTNJtrGEEH6/aOL8V9ogwwZuIvb/TEidCkfcf\nzg/pTcGf2My0MiJLk47xO6EgzNdso9mMG5ZYPraBBsuo7NupvWxCp7NyCiOJDqGH\nK5NmhjInjzsjTghIQRq5+qcCgYEAxnm/NjjvslL8F69p/I3cDJ2/RpaG0sMXvbrO\nVWaMryQyWGz9OfNgGIbeMu2Jj90dar6ChcfUmb8lGOi2AZl/VGmc/jqaMKFnElHl\nJ5JyMFicUzPMiG8DBH+gB71W4Iy+BBKwugHBQP2hkytewQ++PtKuP+RjADEz6vCN\n0mv0WS8CgYBnbMRP8wIOLJPRMw/iL9BdMf606X4xbmNn9HWVp2mH9D3D51kDFvls\n7y2vEaYkFv3XoYgVN9ZHDUbM/YTUozKjcAcvz0syLQb8wRwKeo+XSmo09+360r18\nzRugoE7bPl39WdGWaW3td0qf1r9z3sE2iWUTJPRQ3DYpsLOYIgyKmw==\n\n"

I just removed -----BEGIN CERTIFICATE-----, -----END CERTIFICATE-----, -----BEGIN RSA PRIVATE KEY-----, -----END RSA PRIVATE KEY-----.

Now it complains:

2022-06-10T06:02:27.632844455Z 2022/06/10 06:02:27 [error] 56#56: *14 [lua] config_yaml.lua:222: failed to check item data of [upstreams] err:failed to parse cert: PEM_read_bio_X509_AUX() failed ,val: {"hash_on":"vars","scheme":"grpcs","id":1,"pass_host":"pass","type":"roundrobin","nodes":{"httpbin.org:80":1},"tls":{"client_key":"\nMIIEpAIBAAKCAQEAnyiPLohB\/4n2YpEp0Wt\/xNgeKIVVkcI6PyMcgxFqJoEeLS5N\naUiYT\/+Egi1rjEExVk20qrdSBWMuGW1Uhx8hqDT5iRqx0SQhhPrIKX859Bo1eJV0\nDyQ9JOhkdQl9jKNU1nRakifx3OQEMHEBZz36CwMLAcuMqq5Zn\/emQFMrZf+2ZI3+\nD+5iZCR7TP1oEkdKRoY2UwBkX+QyVqDudZIt4tySPteZjoZp5wqZ5LJxlT35fdqv\ndh8\/+L94qhPlE4T2EaXBm53XczLz2gl4mr4PAf7ti1W5+JdGnWpqkBnqTgIw\/9ca\n2jlT9lttltD87Q1yeKy3vnGqTUuKBrklH5CBDQIDAQABAoIBAHDe5bPdQ9jCcW3z\nfpGax\/DER5b6\/\/UvpfkSoGy\/E+Wcmdb2yEVLC2FoVwOuzF+Z+DA5SU\/sVAmoDZBQ\nvapZxJeygejeeo5ULkVNSFhNdr8LOzJ54uW+EHK1MFDj2xq61jaEK5sNIvRA7Eui\nSJl8FXBrxwmN3gNJRBwzF770fImHUfZt0YU3rWKw5Qin7QnlUzW2KPUltnSEq\/xB\nkIzyWpuj7iAm9wTjH9Vy06sWCmxj1lzTTXlanjPb1jOTaOhbQMpyaAzRgQN8PZiE\nYKCarzVj7BJr7\/vZYpnQtQDY12UL5n33BEqMP0VNHVqv+ZO3bktfvlwBru5ZJ7Cf\nURLsSc0CgYEAyz7FzV7cZYgjfUFD67MIS1HtVk7SX0UiYCsrGy8zA19tkhe3XVpc\nCZSwkjzjdEk0zEwiNAtawrDlR1m2kverbhhCHqXUOHwEpujMBjeJCNUVEh3OABr8\nvf2WJ6D1IRh8FA5CYLZP7aZ41fcxAnvIPAEThemLQL3C4H5H5NG2WFsCgYEAyHhP\nonpS\/Eo\/OXKYFLR\/mvjizRVSomz1lVVL+GWMUYQsmgsPyBJgyAOX3Pqt9catgxhM\nDbEr7EWTxth3YeVzamiJPNVK0HvCax9gQ0KkOmtbrfN54zBHOJ+ieYhsieZLMgjx\niu7Ieo6LDGV39HkvekzutZpypiCpKlMaFlCFiLcCgYEAmAgRsEj4Nh665VPvuZzH\nZIgZMAlwBgHR7\/v6l7AbybcVYEXLTNJtrGEEH6\/aOL8V9ogwwZuIvb\/TEidCkfcf\nzg\/pTcGf2My0MiJLk47xO6EgzNdso9mMG5ZYPraBBsuo7NupvWxCp7NyCiOJDqGH\nK5NmhjInjzsjTghIQRq5+qcCgYEAxnm\/NjjvslL8F69p\/I3cDJ2\/RpaG0sMXvbrO\nVWaMryQyWGz9OfNgGIbeMu2Jj90dar6ChcfUmb8lGOi2AZl\/VGmc\/jqaMKFnElHl\nJ5JyMFicUzPMiG8DBH+gB71W4Iy+BBKwugHBQP2hkytewQ++PtKuP+RjADEz6vCN\n0mv0WS8CgYBnbMRP8wIOLJPRMw\/iL9BdMf606X4xbmNn9HWVp2mH9D3D51kDFvls\n7y2vEaYkFv3XoYgVN9ZHDUbM\/YTUozKjcAcvz0syLQb8wRwKeo+XSmo09+360r18\nzRugoE7bPl39WdGWaW3td0qf1r9z3sE2iWUTJPRQ3DYpsLOYIgyKmw==\n\n","client_cert":"\nMIIDOjCCAiICAwD6zzANBgkqhkiG9w0BAQsFADBnMQswCQYDVQQGEwJjbjESMBAG\nA1UECAwJR3VhbmdEb25nMQ8wDQYDVQQHDAZaaHVIYWkxDTALBgNVBAoMBGFwaTcx\nDDAKBgNVBAsMA29wczEWMBQGA1UEAwwNY2EuYXBpc2l4LmRldjAeFw0yMDA2MjAx\nMzE1MDBaFw0zMDA3MDgxMzE1MDBaMF0xCzAJBgNVBAYTAmNuMRIwEAYDVQQIDAlH\ndWFuZ0RvbmcxDTALBgNVBAoMBGFwaTcxDzANBgNVBAcMBlpodUhhaTEaMBgGA1UE\nAwwRY2xpZW50LmFwaXNpeC5kZXYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK\nAoIBAQCfKI8uiEH\/ifZikSnRa3\/E2B4ohVWRwjo\/IxyDEWomgR4tLk1pSJhP\/4SC\nLWuMQTFWTbSqt1IFYy4ZbVSHHyGoNPmJGrHRJCGE+sgpfzn0GjV4lXQPJD0k6GR1\nCX2Mo1TWdFqSJ\/Hc5AQwcQFnPfoLAwsBy4yqrlmf96ZAUytl\/7Zkjf4P7mJkJHtM\n\/WgSR0pGhjZTAGRf5DJWoO51ki3i3JI+15mOhmnnCpnksnGVPfl92q92Hz\/4v3iq\nE+UThPYRpcGbnddzMvPaCXiavg8B\/u2LVbn4l0adamqQGepOAjD\/1xraOVP2W22W\n0PztDXJ4rLe+capNS4oGuSUfkIENAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAHKn\nHxUhuk\/nL2Sg5UB84OoJe5XPgNBvVMKN0c\/NAPKVIPninvUcG\/mHeKexPzE0sMga\nRNos75N2199EXydqUcsJ8jL0cNtQ2k5JQXXg0ntNC4tuCgIKAOnO879y5hSG36e5\n7wmAoVKnabgjej09zG1kkXvAmpgqoxeVCu7h7fK+AurLbsGCTaHoA5pG1tcHDxJQ\nfpVcbBfwQDSBW3SQjiRqX453\/01nw6kbOeLKYraJysaG8ZU2K8+WpW6JDubciHjw\nfQnpU2U16XKivhxeuKYrV\/INL0sxj\/fZraNYErvJWzh5llvIdNLmeSPmvb50JUIs\n+lDqn1MobTXzDpuCFXA=\n\n"}}, context: init_worker_by_lua*

BFergerson avatar Jun 10 '22 06:06 BFergerson

Now it complains:

Yes, I know, it's expected exception (because of formatting issues) and I'm looking for available ways to configure it, let me think about it.

tzssangglass avatar Jun 10 '22 06:06 tzssangglass

hi @BFergerson , verified and currently unable to do so. The reasons are as follows

  1. if CCERT has -----BEGIN CERTIFICATE----- then it would be env CCERT=-----BEGIN CERTIFICATE----- in nginx.conf, this format is considered incorrect by nginx.
  2. if CCERT doesn't have -----BEGIN CERTIFICATE-----, this format is considered incorrect by https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/ssl.md#parse_pem_cert.

it looks like that you can only set cert and key in apisix.yaml

tzssangglass avatar Jun 10 '22 06:06 tzssangglass

@tzssangglass, what do you think the level of effort would be to support being able to set these values via files? It's been brought up before https://github.com/apache/apisix/discussions/5562.

I think it would be a useful feature and I'm willing to give it a shot. Right now it looks like you have to either hardcode certificates or use an external scripting solution.

Any direction you could offer would be helpful. My plan would be to add the config keys:

  • upstreams[].tls.client_cert_file
  • upstreams[].tls.client_key_file
  • ssl[].cert_file
  • ssl[].key_file

BFergerson avatar Jun 10 '22 08:06 BFergerson

I think it would be a useful feature and I'm willing to give it a shot. Right now it looks like you have to either hardcode certificates or use an external scripting solution.

I tried to implement it with some hacking, but didn't find a usable way. Do you have any ideas? cc @spacewander @tokers @membphis

Maybe we can add functionality to support reading cert and key in standalone mode.

tzssangglass avatar Jun 10 '22 10:06 tzssangglass

We are adding a way to fetch TLS bundles in Upstream objects through the SSL objects. I think adjusting SSL object will be more appropriate.

tokers avatar Jun 12 '22 06:06 tokers

By the way, have you ever tried to escape the linebreak?

tokers avatar Jun 12 '22 06:06 tokers

We are adding a way to fetch TLS bundles in Upstream objects through the SSL objects. I think adjusting SSL object will be more appropriate.

Where can I follow the progress of this work?

By the way, have you ever tried to escape the linebreak?

The new lines aren't the issue. It looks like it's just having spacing in environment variables. The following also fails:

ENV CCERT="test test"
ENV CKEY="test test"

BFergerson avatar Jun 12 '22 07:06 BFergerson

@tzssangglass Then let's try to use double quotes to wrap the env values?

tokers avatar Jun 13 '22 01:06 tokers

Tried to be clever with:

upstreams:
  -
    id: 1
    nodes:
      "httpbin.org:80": 1
    type: roundrobin
    scheme: grpcs
    tls:
      client_cert: -----BEGIN CERTIFICATE-----${{CCERT}}-----END CERTIFICATE-----
      client_key: -----BEGIN RSA PRIVATE KEY-----${{CKEY}}-----END RSA PRIVATE KEY-----
#END

But / is read as \/.

So the following:

ENV CCERT="/test/test"

Is loaded as:

"client_cert":"-----BEGIN CERTIFICATE-----\/test\/test-----END CERTIFICATE-----"

BFergerson avatar Jun 14 '22 15:06 BFergerson

@tzssangglass Then let's try to use double quotes to wrap the env values?

Tried and failed

tzssangglass avatar Jun 14 '22 15:06 tzssangglass

We are adding a way to fetch TLS bundles in Upstream objects through the SSL objects. I think adjusting SSL object will be more appropriate.

Is there a way I can follow the progress of this work?

BFergerson avatar Jun 22 '22 05:06 BFergerson

How about use Cert_ID instead like this? @BFergerson

ssls:
    -
        id: 1
        cert: |
            -----BEGIN CERTIFICATE-----
            cert
            -----END CERTIFICATE-----
        key: |
            -----BEGIN PRIVATE KEY-----
            key
            -----END PRIVATE KEY-----
        snis:
            - "yourdomain.com"

upstreams:
  -
    id: 1
    nodes:
      "httpbin.org:80": 1
    type: roundrobin
    scheme: grpcs
    tls:
      client_cert_id: ${{CERT_ID}}

export CERT_ID=1

monkeyDluffy6017 avatar Jan 12 '23 06:01 monkeyDluffy6017

Hey all,

After some digging I managed to create the following hotfix which patches the apisix/cli/ops.lua.

sed -i
  -e '635i sys_conf["envs"]= {}'
  -e 's~table_insert(sys_conf\["envs"\], name .. "=" .. value)~table_insert(sys_conf["envs"], name)~g'
  /usr/local/apisix/apisix/cli/ops.lua

With the above patch we fix 2 problems.

  1. There are values in sys_conf["envs"] that don't contain the environment variable name but only the value. This is patched by clearing sys_conf["envs"] on line 635.
  2. Environment variable values are not properly escaped. This issue is patched by not setting the value and just letting nginx load the value from the environment.

I hope this helps someone with more lua skills to write a testcase and a fix for this show stopping issue.

boekkooi-lengoo avatar Jan 12 '23 10:01 boekkooi-lengoo

Definitely this is a stopper in some cases. Why do we want to set the environment variable in the nginx configuration file if we want to read it from the shell? It does not make sense unless having a default value that needs to be defined somewhere.

szczad avatar Jul 05 '23 13:07 szczad

This issue has been marked as stale due to 350 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

github-actions[bot] avatar Jun 20 '24 10:06 github-actions[bot]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.

github-actions[bot] avatar Jul 04 '24 10:07 github-actions[bot]

The issue is still there, can't set certificates via env using docker compose. Is there an official fix coming?

NikBelyaev avatar Mar 21 '25 12:03 NikBelyaev

Just added to https://github.com/orgs/apache/projects/273

juzhiyuan avatar Mar 24 '25 01:03 juzhiyuan