Renew certificate when expire

[abozhinov]

Hi, I have a problem when certificate is expired saved in Redis.

We have multi-tenant app and user Redis for storage service. The problem I met today is that Nginx serve expired certificates.

We use AWS and has autoscaling group. I delete all certificates from Redis and Nginx continue to serve expired certificates and didn't create new certs.

When create new instance in autoscaling group NGINX start create new certificates. May be the problem is in local cache.

lua_shared_dict auto_ssl 5m; lua_shared_dict auto_ssl_settings 64k; ............ auto_ssl = (require "resty.auto-ssl").new({ storage_adapter = "resty.auto-ssl.storage_adapters.redis", redis = { host = os.getenv("DREAMSHOP__REDIS__HOST"), port = 6379, prefix = os.getenv("DREAMSHOP__REDIS__CERTS__KEYS__PREFIX") }, allow_domain = function(domain, allow_domain_auto_ssl)

    if ngx.re.match(domain, "^(.*mydreamshop.io).*$", "ijo") then
        return false
    end

    local redis = allow_domain_auto_ssl.storage.adapter:get_connection()
    local prefix = allow_domain_auto_ssl:get("sites_prefix")

    if redis then
        if 1 == redis:exists(prefix.."["..domain.."][1]") then
           return true
        else
           return false
        end
    else
        return true
    end
end

})

auto_ssl:set("sites_prefix", os.getenv("DREAMSHOP__REDIS__DOMAINS__KEYS__PREFIX")) auto_ssl:init()

[did]

hi @abozhinov, how did you solve your issue? We're facing the same problem here. Besides, we've got nothing in Nginx logs. Thanks!

[abozhinov]

Hi, the problem was related to the cache folder where save all certs. You should delete the cache folder every day.

[andreasschroth]

@abozhinov You mean the folder where all the certs are saved? Why would you want to delete that folder (and thus all certificates as well) everyday?

[abozhinov]

Because auto-ssl continue to serve them from the cache folder. You need to make something like cronjob that will empty this folder every day or something like that. We have autoscaling group and we face this problem only ones.

[did]

hey @abozhinov, thanks for the answer. Actually, what you said is one part of the issue we got.

Anyway, I'm posting our issue here in case someone meets it. What happens is that we changed our redis LRU cache strategy from no_eviction to allkeys-lru. So Redis started to delete certificates but not properly meaning that we ended up with only key for a single domain (instead of 2 in a normal situation as we noticed: domain.com:latest and domain.com:). That confused auto-ssl which then became unable to generate a new certificate to replace expired certificates. Removing the one key left for a domain + restarting Nginx didn't fix the problem. We had to restart the whole server. Now, thanks to you I understand why rebooting worked (it erased the cache folder).

So, here is the conclusion we came up: auto-ssl has to use its own Redis instance set up with the no_eviction LRU cache strategy (+ backup).

[itshikanov]

Because auto-ssl continue to serve them from the cache folder. You need to make something like cronjob that will empty this folder every day or something like that. We have autoscaling group and we face this problem only ones.

The Cache folder is /etc/resty-auto-ssl/letsencrypt/certs ?

[abozhinov]

The problem is not in the CACHE folder :) the problem is in the Redis. Nginx uses the old cert because we still have them in the REDIS.

[dbackeus]

We have been running autoresty + lua-resty-auto-ssl for almost a year now without problems but for some reason during the past 4 days already existing domains have not been renewed.

The only fix we've found is to:

1. Delete the certificates from disk
2. Delete the certificates from redis
3. Restart openresty
4. Make a request to the expired domain to re-issue a new working certificate

I'm quite confused as to why renewal stopped happening in the first place (no errors found in the logs). Also confused regarding how the disk storage is actually used. Is clearing it out daily really the right solution? At the end of the day feels like there must be some bug related to checking for expiry in the lua script?

Also, maybe off topic, but why has it been more than a year since the last release of this script considering how many commits and pull request have been merged in the meantime? I noticed some commits related to expiring old domains so I wonder if this behaviour might even be fixed in HEAD...

[ronaldgrn]

Hey @dbackeus did you ever get a workaround for this?

[dbackeus]

We found that the reason certificate regeneration broke was that we had added frequent reload signals to our openresty instance via crontab. We did this since we found that our instances would randomly become unresponsive after a few days or weeks of runtime. However it turns out that the reload signal also reset the time interval lue-resty-auto-ssl uses to look for outdated certificates. So effectively the time interval was never reached so certificates would never expire.

We settled for the following solution...

Added to sudo crontab -e:

# Reload openresty once per day to mitigate unexpected downtime events.
0 0 * * * /etc/init.d/openresty reload > /var/log/openresty/cron.log

Added to init_by_lua_block in /etc/openresty/nginx.conf:

-- Set the certificate refresh at 20 hours to allow us to reload nginx once
-- per day without breaking the refresh.
auto_ssl:set("renew_check_interval", 72000)

@ronaldgrn hope this helps!

[waynegemmell]

I have a similar problem. We don't restart openresty so I don't think it's exactly the same. It would be nice to get an official response on this.