lua-resty-auto-ssl
lua-resty-auto-ssl copied to clipboard
auto-ssl
Error generating new certificates
Hi.. just notice today that it started to give errors on generating/renewing certificates.. upgraded openresty and the luarocks dependencies. however is still gives errors: 2019/12/06 11:23:46 [error] 6634#6634: 337 [lua] lets_encrypt.lua:51: issue_cert(): auto-ssl: error fetching certificate from storage for : closed, context: ssl_certificate_by_lua, client: , server: 0.0.0.0:443 2019/12/06 11:23:46 [error] 6634#6634: *337 [lua] lets_encrypt.lua:77: issue_cert(): auto-ssl: dehydrated manual hook.sh failed: env HOOK_SECRET=371ac024d834862be24sd79bd1dd941fcsa0c4f191037692493f156sdffa3f4c28 HOOK_SERVER_PORT=8999 /usr/local/openresty/luajit/bin/resty-auto-ssl/letsencrypt_hooks deploy_cert c.led.ad /etc/resty-auto-ssl/letsencrypt/certs/c.led.ad/privkey.pem /etc/resty-auto-ssl/letsencrypt/certs/c.led.ad/cert.pem /etc/resty-auto-ssl/letsencrypt/certs/c.led.ad/fullchain.pem /etc/resty-auto-ssl/letsencrypt/certs/c.led.ad/chain.pem 1575631426 status: 256 out: deploy_cert failed to get the expiry date did i miss the need for some new conf?
i have upgraded to the latest (0.13.1) also upgraded openresty,dehydrated(and his luarocks dependencies) to the latest.. we then look more carefully and see that although we get this error.. all is working.. the ssl certificates are "extracted" from lets encrypt and placed in redis and served correctly. we did some debugging and come to the conclusion that this is something that is caused by the asynchronous operations.. we have a bad network, some operation takes some time.. causing for this error to appear..
We're also experiencing a similar issue, although the symptom looks different. In our case we're generating several certificates every day, but for some reason sometimes we're facing this issue when trying to generate a new certificate (we haven't still identified a pattern). This is the full error we're getting when it's failing to generate a new SSL certificate:
2020/06/30 07:51:32 [error] 22#22: *18639769 lua entry thread aborted: runtime error: ...sty/luajit/share/lua/5.1/resty/auto-ssl/servers/hook.lua:44: assertion failed!
stack traceback:
coroutine 0:
[C]: in function 'assert'
...sty/luajit/share/lua/5.1/resty/auto-ssl/servers/hook.lua:44: in function 'server'
.../local/openresty/luajit/share/lua/5.1/resty/auto-ssl.lua:95: in function 'hook_server'
content_by_lua(nginx.conf:365):2: in main chunk, client: 127.0.0.1, server: , request: "POST /deploy-cert HTTP/1.1", host: "127.0.0.1:8999"
2020/06/30 07:51:32 [error] 22#22: *18639705 [lua] lets_encrypt.lua:77: issue_cert(): auto-ssl: dehydrated manual hook.sh failed: env HOOK_SECRET=XXXXXXXXX HOOK_SERVER_PORT=8999 /usr/local/openresty/luajit/bin/resty-auto-ssl/letsencrypt_hooks deploy_cert myawesomedomain.com /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/privkey.pem /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/cert.pem /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/fullchain.pem /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/chain.pem 1593503492 status: 256 out: deploy_cert
err: curl: (22) The requested URL returned error: 500 Internal Server Error
hook request (deploy_cert) failed
, context: ssl_certificate_by_lua*, client: 95.169.228.179, server: 0.0.0.0:443
2020/06/30 07:51:32 [error] 22#22: *18639705 [lua] ssl_certificate.lua:97: issue_cert(): auto-ssl: issuing new certificate failed: dehydrated failure, context: ssl_certificate_by_lua*, client: 95.169.228.179, server: 0.0.0.0:443
2020/06/30 07:51:32 [error] 22#22: *18639705 [lua] ssl_certificate.lua:291: auto-ssl: could not get certificate for myawesomedomain.com - using fallback - failed to get or issue certificate, context: ssl_certificate_by_lua*, client: 95.169.228.179, server: 0.0.0.0:443
In our case, we're just storing the SSL certificates in a NFS volume, where we've 2 servers behind a load balancer processing the requests and mounting the NFS volume. Checking the directory /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/ looks like it has already generated some certificates:
root@localhost:/srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com# ls -lrt
total 276
-rw------- 1 www-data www-data 3243 May 25 20:41 privkey-1590439289.pem
-rw------- 1 www-data www-data 1675 May 25 20:41 cert-1590439289.csr
-rw------- 1 www-data www-data 0 May 25 20:41 cert-1590439289.pem
-rw------- 1 www-data www-data 3247 May 25 20:41 privkey-1590439305.pem
-rw------- 1 www-data www-data 1675 May 25 20:41 cert-1590439305.csr
-rw------- 1 www-data www-data 0 May 25 20:41 cert-1590439305.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439320.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439320.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439320.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439325.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439325.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439325.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439335.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439335.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439335.pem
-rw------- 1 www-data www-data 3247 May 25 20:42 privkey-1590439349.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439349.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439349.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439350.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439350.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439350.pem
-rw------- 1 www-data www-data 3247 May 25 20:42 privkey-1590439355.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439355.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439355.pem
-rw------- 1 www-data www-data 3247 May 25 20:42 privkey-1590439364.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439364.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439364.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439365.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439365.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439365.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439368.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439368.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439368.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439370.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439370.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439370.pem
-rw------- 1 www-data www-data 3247 May 25 20:42 privkey-1590439372.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439372.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439372.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439374.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439374.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439374.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439377.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439377.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439377.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439379.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439379.csr
-rw------- 1 www-data www-data 0 May 25 20:42 cert-1590439379.pem
-rw------- 1 www-data www-data 3243 May 25 21:36 privkey-1590442609.pem
-rw------- 1 www-data www-data 1675 May 25 21:36 cert-1590442609.csr
-rw------- 1 www-data www-data 0 May 25 21:36 cert-1590442609.pem
-rw------- 1 www-data www-data 3243 May 25 21:36 privkey-1590442612.pem
-rw------- 1 www-data www-data 1675 May 25 21:36 cert-1590442612.csr
-rw------- 1 www-data www-data 0 May 25 21:36 cert-1590442612.pem
-rw------- 1 www-data www-data 3243 May 25 21:36 privkey-1590442614.pem
-rw------- 1 www-data www-data 1675 May 25 21:36 cert-1590442614.csr
-rw------- 1 www-data www-data 0 May 25 21:36 cert-1590442614.pem
-rw------- 1 www-data www-data 3243 May 25 21:36 privkey-1590442618.pem
-rw------- 1 www-data www-data 1675 May 25 21:36 cert-1590442618.csr
-rw------- 1 www-data www-data 0 May 25 21:36 cert-1590442618.pem
-rw------- 1 www-data www-data 3243 May 25 21:42 privkey-1590442973.pem
-rw------- 1 www-data www-data 1675 May 25 21:42 cert-1590442973.csr
-rw------- 1 www-data www-data 2277 May 25 21:43 cert-1590442973.pem
-rw------- 1 www-data www-data 1648 May 25 21:43 chain-1590442973.pem
lrwxrwxrwx 1 www-data www-data 20 May 25 21:43 chain.pem -> chain-1590442973.pem
lrwxrwxrwx 1 www-data www-data 24 May 25 21:43 fullchain.pem -> fullchain-1590442973.pem
lrwxrwxrwx 1 www-data www-data 19 May 25 21:43 cert.csr -> cert-1590442973.csr
lrwxrwxrwx 1 www-data www-data 19 May 25 21:43 cert.pem -> cert-1590442973.pem
-rw------- 1 www-data www-data 3925 May 25 21:43 fullchain-1590442973.pem
And the only way to recover from this error is just deleting the directory /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com and then forcing new request on the domain looks like the certificate is generated correctly. This is the version we're using at the moment:
lua-resty-auto-ssl
0.13.1-1 (installed) - /usr/local/openresty/luajit/lib/luarocks/rocks-5.1
Any idea of what might cause this problem?
hi.. i would say that this error was about again about how dehydrated works in a asynchronous way and the issue is like internal.. ours is about a "bad" network your is probably related with the fact that you use SSL certificates in a NFS volume, where we've 2 servers behind a load balancer processing the requests and mounting the NFS volume.. that is probably causing delays that in turn causes the ssl validation to fail.. we simply ignored because we couldn't do anything about..
Hi @ruiluis thanks for your comment! It could be the same situation. The fact of purging the certificate directory solves the issue, makes me thing that's something else in our case. What calls my attention is that 500 error from dehydrated, which makes me thing this is caused by some other internal error (I've checked file permissions and looks good):
2020/06/30 07:51:32 [error] 22#22: *18639705 [lua] lets_encrypt.lua:77: issue_cert(): auto-ssl: dehydrated manual hook.sh failed: env HOOK_SECRET=XXXXXXXXX HOOK_SERVER_PORT=8999 /usr/local/openresty/luajit/bin/resty-auto-ssl/letsencrypt_hooks deploy_cert myawesomedomain.com /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/privkey.pem /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/cert.pem /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/fullchain.pem /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/chain.pem 1593503492 status: 256 out: deploy_cert
err: curl: (22) The requested URL returned error: 500 Internal Server Error
In our case we're using EFS and all our infra is in AWS and we haven't detected any poor network performance.