Proxy SSL certificate renewal needs kong reload to take effect

[hnajib-sym]

Is there an existing issue for this?

• [X] I have searched the existing issues

Kong version ($ kong version)

Kong Enterprise 3.1.0.0

Current Behavior

Kong proxy ssl certificate set in kong.conf get renewed automatically by cert-manager , but kong proxy keep serving old certificate unless it restarted or reloaded

Called only once in kong/init.lua

  if config.proxy_ssl_enabled or config.stream_ssl_enabled then
    certificate.init()
  end

Expected Behavior

The proxy ssl certificate should be updated when it's changed

Steps To Reproduce

1. Install cert-manager and setup an issuer ( e.g letsencrypt )
2. Create a certificate in kong namespace ( e.g kong-proxy-cert)
3. Install kong helm chart with override.yaml

   secretVolumes:
   - kong-proxy-cert
   env:
     ssl_cert: /etc/secrets/kong-proxy-cert/tls.crt
     ssl_cert_key: /etc/secrets/kong-proxy-cert/tls.key

   helm install kong-release kong/kong  -f ./ override.yaml --version=2.14.0

5. Check kong proxy ingress certificate

curl  -vv https://<kong-proxy-ingress> 2>&1  | grep -E  "start|expire"

6. Renew the certificate using

cmctl renew -n kong  kong-proxy-cert

7. Check again kong proxy ingress certificate
8. Verify that certificate didn't get updated
9. Run kong reload inside all kong proxy pod ( or rollout restart the deployment )
10. Re-check kong proxy ingress certificate
11. Verify that certificate got updated this time

Anything else?

No response

[ghost]

if you deploy Kong with DB-less deployment, see: https://github.com/Kong/charts/blob/main/charts/kong/README.md#db-less-deployment

there’s no good way to rotate ssl certs, since pushing config is entirely up to the controller, and there’s no designated way to manually trigger a push. link: https://discuss.konghq.com/t/how-to-rotate-mtls-certificates-for-upstream-service/7864/2

[hnajib-sym]

Nginx ingress controller is able to do this check here , I don't seee why it can't be done in kong runloop

[hbagdi]

We recommend using Kong Ingress Controller for automating with k8s and k8s-oriented tooling like cert-manager.

If that is not an option for you, please use Kong's Admin API to install the default proxy certificate. You can do so by creating a Certificate resource in Kong via Kong's Admin API and then associating an SNI resource with name as *. See this PR that adds this feature.

Reloading Kong automatically on file changes can lead to unexpected issues so we don't do it and we have not plans to do that.

[hnajib-sym]

We are already using kong ingress controller but since ssl termination is handled by the proxy there is not much to tweak on KIC .

Loading proxy through the Admin API is not feasible the deployment will fails without a certificate .

Is there an API endpoint for reloading kong similar to kong reload commad line ?

Other options will be through a Plugin , but not sure if possible.

[hbagdi]

@Kong/team-k8s Is there a way to configure the fallback TLS certificate using a CR in KIC?

[rainest]

There's no need for a CR; the configuration in the OP is correct for the fallback certificate.

In general I'd recommend against using the fallback certificate. Proxy certificates should be managed through their Ingress's configuration. Non-fallback certificates are update on the fly, and cert-manager should be able to manage Ingress-associated certificates as well as the fallback.

There isn't a mechanism to get running instances to load a new fallback if the mounted Secret changes and updates the files in the container FS. Gateway could add something to watch those files and trigger a reload if they change, but I wouldn't expect that to get prioritized. Expected usage is that you use Ingress certs, not the fallback. The fallback is handled by NGINX code, so that's not somewhere I'd expect us to add new functionality.

[hbagdi]

There's no need for a CR; the configuration in the OP is correct for the fallback certificate.

The fallback certificate can be configured using the Admin API of Kong via a special SNI value of *. AFAIK, there isn't a way to configure this via Ingress or any of the existing CRs. Should it not be possible to configure the fallback certificate via a k8s resource, perhaps Gateway API has an answer? Perhaps we can (ab)use Ingress to do that as well?

What I was hoping was for a mechanism that gives KIC enough information so that it can dynamically configure the fallback certificate via the API rather than asking the user to deal with mounting secrets and reloading Kong.

[rainest]

Ingress and Gateway both have proscribed means of configuring certificates, and expected usage is that you use them. Our fallback cert is a vendor quirk and should not be something we recommend as standard configuration.

Gateway API doesn't really provide for a fallback certificate as a defined concept, but AFAIK https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.Listener Hostname does let you specify * if you really want to (you shouldn't, please, at least provide some sort of *.example.com instead) if you so desire.

[hnajib-sym]

As per API doc , there still no guarantee that the certificate will reflect the k8s resource when updated

[StarlightIbuki]

Dear contributor, We're closing this issue as there hasn't been any update to it for a long time. If the issue is still relevant in the latest version, please feel free to reopen it. We're more than happy to revisit it again. Your contribution is greatly appreciated! Please have a look at our pledge to the community for more information. Sincerely, Kong Gateway Team