cloud-on-k8s icon indicating copy to clipboard operation
cloud-on-k8s copied to clipboard

Document how to manually rotate auto generated certificates

Open idanmo opened this issue 4 years ago • 8 comments

ECK operator has configuration options that allow users to specify when certificates should be auto-rotated. In some cases users may want to manually trigger auto generated certificates rotation.

Manually triggering these certs rotation can be done by deleting the Kubernetes secrets that are holding them. We should document this method.

idanmo avatar Jul 22 '21 13:07 idanmo

Hi @idanmo we are having an issue where we failed to specify the length of time that the CA should live for. And I think one of the solutions is to force it to generate a new CA and Cert. We need to get it to regen the cert but trust both the old one and the new one that it creates so that logstash can communicate over TLS without service interruption when it does the role. I have tried turning off TLS verify, but once the CA has rolled even that doesn't work. Any advice on how to execute this cleanly??

asazallesmilner avatar Sep 13 '21 22:09 asazallesmilner

@asazallesmilner You can modify the value of field ca-cert-validity of configuration file eck.yaml

ghost avatar Sep 15 '21 15:09 ghost

Manually triggering these certs rotation can be done by deleting the Kubernetes secrets that are holding them.

Is it something that could be confirmed officially ? @sebgl

mtparet avatar Jan 20 '22 13:01 mtparet

Manually triggering these certs rotation can be done by deleting the Kubernetes secrets that are holding them.

Is it something that could be confirmed officially ? @sebgl

Yes.

sebgl avatar Jan 20 '22 16:01 sebgl

Hi, I'm running ECK v.1.2.1 and I was trying to extend the validity of my certs by changing the ca-cert-validity and cert-validity config options. I then deleted one of the secrets (kibana-eck-c1-kb-http-certs-public) to trigger cert rotation/regeneration and it was recreated but it seems that the validation time is not changed at all. Is it something that I'm missing here?

hreidar avatar Nov 11 '22 09:11 hreidar

@sebgl Hi! Please, help. I met the situation when deleting the certificates don't cause its rotating. Is there the instruction how to manually to renew certs? Or may be You can give the advice with correct order of deleting.. after certs expiring/renew period came, not all certs renewed (kibana does not), how to renew them in such situation? ECK 2.0.0 eck config

   ...
    ca-cert-validity: 8760h
    ca-cert-rotate-before: 24h
    cert-validity: 8760h
    cert-rotate-before: 24h
  ...

I have such secrets with certificates :

# secrets where certs were renewed:
elasticsearch-es-remote-ca
elasticsearch-es-transport-ca-internal
elasticsearch-es-transport-certs-public
# secrets, where certs were not renewed, even after deleting:  
heartbeat-beat-es-ca
kibana-kb-es-ca
metricbeat-beat-es-ca
# secrets where certs were also not renewd, but seems they have other management mechanics : 
elasticsearch-es-http-ca-internal
elasticsearch-es-http-certs-internal
elasticsearch-es-http-certs-public

usefree avatar Nov 10 '23 12:11 usefree

i found issue in release notes https://github.com/elastic/cloud-on-k8s/issues/5621 and realised that in order to renew secrets :

# secrets, where certs were not renewed, even after deleting:  
heartbeat-beat-es-ca
kibana-kb-es-ca
metricbeat-beat-es-ca

i should delete secrets :

elasticsearch-es-http-ca-internal
elasticsearch-es-http-certs-internal
elasticsearch-es-http-certs-public

after that all certs updated

usefree avatar Nov 10 '23 13:11 usefree

This is helpful. thanks.

I was trying to extend the validity of my certs by changing the ca-cert-validity and cert-validity config options.

I was also considering this but I've ended up using the default 1y validity period and use this helper service instead https://github.com/stakater/Reloader to automatically restart some pods (which are clients for Elasticsearch) that I maintain when the TLS and/or the User/Password secrets are updated by ECK.

    secret.reloader.stakater.com/reload: "data-es-http-certs-public,data-es-elastic-user"

This is handy for anyone who is having Elasticsearch clients that are not ECK-managed.

christidis avatar Dec 18 '23 13:12 christidis