500 Internal Server Error when deploy_cert

[ivanmp91]

Hi! We're using auto-ssl + openresty to generate several certificates every day, but for some reason sometimes we're facing this issue when trying to generate a new certificate (we haven't still identified a pattern). This is the full error we're getting when it's failing to generate a new SSL certificate:

2020/06/30 07:51:32 [error] 22#22: *18639769 lua entry thread aborted: runtime error: ...sty/luajit/share/lua/5.1/resty/auto-ssl/servers/hook.lua:44: assertion failed!
stack traceback:
coroutine 0:
	[C]: in function 'assert'
	...sty/luajit/share/lua/5.1/resty/auto-ssl/servers/hook.lua:44: in function 'server'
	.../local/openresty/luajit/share/lua/5.1/resty/auto-ssl.lua:95: in function 'hook_server'
	content_by_lua(nginx.conf:365):2: in main chunk, client: 127.0.0.1, server: , request: "POST /deploy-cert HTTP/1.1", host: "127.0.0.1:8999"
2020/06/30 07:51:32 [error] 22#22: *18639705 [lua] lets_encrypt.lua:77: issue_cert(): auto-ssl: dehydrated manual hook.sh failed: env HOOK_SECRET=XXXXXXXXX HOOK_SERVER_PORT=8999 /usr/local/openresty/luajit/bin/resty-auto-ssl/letsencrypt_hooks deploy_cert myawesomedomain.com /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/privkey.pem /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/cert.pem /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/fullchain.pem /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/chain.pem 1593503492 status: 256 out: deploy_cert
 err: curl: (22) The requested URL returned error: 500 Internal Server Error
hook request (deploy_cert) failed
, context: ssl_certificate_by_lua*, client: 95.169.228.179, server: 0.0.0.0:443
2020/06/30 07:51:32 [error] 22#22: *18639705 [lua] ssl_certificate.lua:97: issue_cert(): auto-ssl: issuing new certificate failed: dehydrated failure, context: ssl_certificate_by_lua*, client: 95.169.228.179, server: 0.0.0.0:443
2020/06/30 07:51:32 [error] 22#22: *18639705 [lua] ssl_certificate.lua:291: auto-ssl: could not get certificate for myawesomedomain.com - using fallback - failed to get or issue certificate, context: ssl_certificate_by_lua*, client: 95.169.228.179, server: 0.0.0.0:443

In our case, we're just storing the SSL certificates in a NFS volume, where we've 2 servers behind a load balancer processing the requests and mounting the NFS volume. Checking the directory /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com/ looks like it has already generated some certificates:

root@localhost:/srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com# ls -lrt
total 276
-rw------- 1 www-data www-data 3243 May 25 20:41 privkey-1590439289.pem
-rw------- 1 www-data www-data 1675 May 25 20:41 cert-1590439289.csr
-rw------- 1 www-data www-data    0 May 25 20:41 cert-1590439289.pem
-rw------- 1 www-data www-data 3247 May 25 20:41 privkey-1590439305.pem
-rw------- 1 www-data www-data 1675 May 25 20:41 cert-1590439305.csr
-rw------- 1 www-data www-data    0 May 25 20:41 cert-1590439305.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439320.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439320.csr
-rw------- 1 www-data www-data    0 May 25 20:42 cert-1590439320.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439325.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439325.csr
-rw------- 1 www-data www-data    0 May 25 20:42 cert-1590439325.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439335.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439335.csr
-rw------- 1 www-data www-data    0 May 25 20:42 cert-1590439335.pem
-rw------- 1 www-data www-data 3247 May 25 20:42 privkey-1590439349.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439349.csr
-rw------- 1 www-data www-data    0 May 25 20:42 cert-1590439349.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439350.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439350.csr
-rw------- 1 www-data www-data    0 May 25 20:42 cert-1590439350.pem
-rw------- 1 www-data www-data 3247 May 25 20:42 privkey-1590439355.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439355.csr
-rw------- 1 www-data www-data    0 May 25 20:42 cert-1590439355.pem
-rw------- 1 www-data www-data 3247 May 25 20:42 privkey-1590439364.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439364.csr
-rw------- 1 www-data www-data    0 May 25 20:42 cert-1590439364.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439365.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439365.csr
-rw------- 1 www-data www-data    0 May 25 20:42 cert-1590439365.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439368.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439368.csr
-rw------- 1 www-data www-data    0 May 25 20:42 cert-1590439368.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439370.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439370.csr
-rw------- 1 www-data www-data    0 May 25 20:42 cert-1590439370.pem
-rw------- 1 www-data www-data 3247 May 25 20:42 privkey-1590439372.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439372.csr
-rw------- 1 www-data www-data    0 May 25 20:42 cert-1590439372.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439374.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439374.csr
-rw------- 1 www-data www-data    0 May 25 20:42 cert-1590439374.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439377.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439377.csr
-rw------- 1 www-data www-data    0 May 25 20:42 cert-1590439377.pem
-rw------- 1 www-data www-data 3243 May 25 20:42 privkey-1590439379.pem
-rw------- 1 www-data www-data 1675 May 25 20:42 cert-1590439379.csr
-rw------- 1 www-data www-data    0 May 25 20:42 cert-1590439379.pem
-rw------- 1 www-data www-data 3243 May 25 21:36 privkey-1590442609.pem
-rw------- 1 www-data www-data 1675 May 25 21:36 cert-1590442609.csr
-rw------- 1 www-data www-data    0 May 25 21:36 cert-1590442609.pem
-rw------- 1 www-data www-data 3243 May 25 21:36 privkey-1590442612.pem
-rw------- 1 www-data www-data 1675 May 25 21:36 cert-1590442612.csr
-rw------- 1 www-data www-data    0 May 25 21:36 cert-1590442612.pem
-rw------- 1 www-data www-data 3243 May 25 21:36 privkey-1590442614.pem
-rw------- 1 www-data www-data 1675 May 25 21:36 cert-1590442614.csr
-rw------- 1 www-data www-data    0 May 25 21:36 cert-1590442614.pem
-rw------- 1 www-data www-data 3243 May 25 21:36 privkey-1590442618.pem
-rw------- 1 www-data www-data 1675 May 25 21:36 cert-1590442618.csr
-rw------- 1 www-data www-data    0 May 25 21:36 cert-1590442618.pem
-rw------- 1 www-data www-data 3243 May 25 21:42 privkey-1590442973.pem
-rw------- 1 www-data www-data 1675 May 25 21:42 cert-1590442973.csr
-rw------- 1 www-data www-data 2277 May 25 21:43 cert-1590442973.pem
-rw------- 1 www-data www-data 1648 May 25 21:43 chain-1590442973.pem
lrwxrwxrwx 1 www-data www-data   20 May 25 21:43 chain.pem -> chain-1590442973.pem
lrwxrwxrwx 1 www-data www-data   24 May 25 21:43 fullchain.pem -> fullchain-1590442973.pem
lrwxrwxrwx 1 www-data www-data   19 May 25 21:43 cert.csr -> cert-1590442973.csr
lrwxrwxrwx 1 www-data www-data   19 May 25 21:43 cert.pem -> cert-1590442973.pem
-rw------- 1 www-data www-data 3925 May 25 21:43 fullchain-1590442973.pem

And the only way to recover from this error is just deleting the directory /srv/domains/letsencrypt/letsencrypt/certs/myawesomedomain.com and then forcing new request on the domain looks like the certificate is generated correctly. This is the version we're using at the moment:

lua-resty-auto-ssl
   0.13.1-1 (installed) - /usr/local/openresty/luajit/lib/luarocks/rocks-5.1

Any idea of what might cause this problem?

[kpagcha]

Did you manage to solve this at all?