Moving from CertificateRequests to Certificates: A Survey

[maelvls]

Dear users of cert-manager/openshift-routes,

I'd like to hear your use-cases of cert-manager/openshift-routes. The initial reason openshift-routes was created was because an internal customer was using cert-utils-operator and didn't like the idea of having the private key in Secret resources. Also, cert-utils-operator doesn't create the certificate out of annotations like what's possible with the Ingress and Gateway resources. This is why cert-manager/openshift-routes doesn't create a Certificate resources.

In https://github.com/cert-manager/openshift-routes/issues/34 and https://github.com/cert-manager/openshift-routes/issues/42, I learned that this choice makes it hard for people to adopt this tool. For example, the project https://github.com/its4u/cert-manager-routes-controller/ does the same thing except it creates a Certificate resource rather than a CertificateRequest.

If you are using cert-manager/openshift-routes, why do you use it? Do you use it as a way to request certificates right from the Route resource?

[nate-duke]

We use it to allow certificate-manager to manage certificates for Routes. We do not use redhat's cert-utils-operator.

We use an in-house ACME system to issue our certs.

[ctml91]

We use it as a way to request certs directly from the route resource, as using ingress adds a bit of unnecessary complexity (OCP users now need to create Ingress for the purpose of creating a Route, when many are only familiar with Routes).

Personally I'd prefer if openshift-routes used Certificate resources, and allow cert-manager to handle using all the logic built into it already. IMO there is no difference in having the PK storage directly in the Route or Secret. Either way, it is being stored in etcd. By default cluster role definitions secrets are more secure than routes, as default cluster roles do not permit users with view privileges in a namespace to view secrets, however the view role permits users to view routes which gives them access to the PK. Either way, the PK will end up viewable in the Route so it's somewhat a moot point I think.

I suppose you could create custom roles to restrict access to the routes if you wanted to, while granting access to secrets, but it seems a bit backwards. But you could also create custom roles to restrict access to both secrets & routes if you truly wanted to restrict a user with access to a namespace from viewing PK's.

I think the benefits of using a certificate resource outweigh any pros of embedding directly in routes to avoid storing PK in Routes. I think if cert-manager is okay with storing PK in secrets, then openshift-routes should too.

[jacksgt]

Hello,

we use openshift-routes as part of our internal PaaS together with Let's Encrypt (for a couple hundred websites). As @ctml91 already pointed out, it's important for us that users can keep using the native OpenShift Route resources (and maybe need to add a couple of annotation), but don't need to rewrite their deployment manifests to use Ingress.

We were previously using the https://github.com/tnozicka/openshift-acme project (long deprecated) for the same purpose, but have switched to openshift-routes because it's an official cert-manager project.

I'd prefer if openshift-routes used Certificate resources, and allow cert-manager to handle using all the logic built into it already. IMO there is no difference in having the PK storage directly in the Route or Secret

I fully agree with this and am heavily in favor of switching to Certificates. As I have already pointed out in https://github.com/cert-manager/openshift-routes/issues/42 , we are currently a bit blind what's going on with the CertfificateRequests created by openshift-routes in our clusters simply because there are no Prometheus metrics available for it. This lead to some expired certificates in the past.

[jacksgt]

I'll put together a PR that provisions Certificates instead of CertificateRequest in the next couple of days. Then we can discuss there what the implications of that are.

[maelvls]

I mentioned https://github.com/cert-manager/openshift-routes/pull/55 during this morning's cert-manager open standup. I thought I understood the benefits of migrating to creating Certificates, but I struggled explaining the reasons. Would it be possible to meet at the next cert-manager community meeting (Thu 7 March 18:00 Paris time)?

Before then, Ill try to gather the pros and cons of each approach.

[ctrought]

Some benefits of moving to Certificates

1. increasing observability #42
2. the certificate will be stored in standard kubernetes secrets, allowing one to use elsewhere if desired. For example, for re-encrypt Routes one may also want to mount that certificate in the pod for the application to use as well. This cannot be done currently with openshift-routes because the cert is being stored directly in the Route object.
   • If using a re-encrypt route right now, one would probably have to generate another cert managed by a different process such as a separate Certificate resource in cert-manager or the built in service certificates managed by OpenShift which will both result in a cert that lives in a secret and can be used by the underlying application as a volume mount
3. it will be simpler to implement sharing of certificates between routes #54, similar to how ingress objects can share the underlying cert-manager generated secret whether the certificate being configured using a Certificate or through annotations on one ingress object. Briefly mentioned in the proposed PR https://github.com/cert-manager/openshift-routes/pull/55#issuecomment-1943313215
4. if 3) is implemented and a route can be pointed at a Certificate object, this will enable the use of all fields in a Certificate even if there are not supported annotations to configure those fields and makes the adoption of any new features much simpler as the logic doesn't need to be duplicated into openshift-routes.
   For example, I may want to have a JKS keystore generated for the backend of my application if using a re-encrypt route along with the PEM format used in the Route, leveraging a Certificate I'll be able to get both formats output. Another example is #46 where the revision history is not currently supported in openshift-routes but it could still be configured if directly using a Certificate object and linking the Route to it, or using a mutation policy to add it to the Certificate resource when generated by openshift-routes using a tool like Gatekeeper/Kyverno. The option of supporting these features in cert-manager routes would still be possible via annotations the same way it is for cert-manager and ingress objects, but only as a means to configure the Certificate resource and any complex logic can be offloaded to cert-manager.
5. it will consistent, native ingress objects and routes are arguably the same thing so why should they not function the same way?

I understand initially there was one internal customer who didn't like the idea of having the private key in Secret resources which was a driver for using CertificateRequests, but as the adoption grows I think this is most likely not a popular opinion of users of openshift-routes. The default aggregated ClusterRole view in Kubernetes does not provide GET permissions to Secrets while in OpenShift it does provide GET access to Routes making it less secure by RBAC standards so there should imo not be any negative of using Secrets.

Personally I think there are some clear benefits of using Certificates to manage the lifecycle of the cert. On the other hand, the benefits of configuring CertificateRequests directly are not very clear imo.

I tried to summarize as best as I could, but please add to this if there are any other use cases I missed.

[maelvls]

Thank you @ctrought, that is super useful, you perfectly laid out the pros and cons. @jacksgt Do you think you can join the dev meeting this afternoon? Thanks!

[maelvls]

Increasing observability Monitoring observability for "CertificateRequests"

Fair enough, that's a good point. But if this is the primary reason for creating Certificates, wouldn't it make more sense for openshift-routes to expose its own Routes metrics that mimic cert-manager's metrics to get the same result?

Sharing of certificates between routes

[Attach multiple routes to the same Certificate] should be possible with an annotation like cert-manager.io/certificate-name, which would solve https://github.com/cert-manager/openshift-routes/issues/54 (https://github.com/cert-manager/openshift-routes/pull/55#issuecomment-1943313215)

How big is this need? Is that something folks do often? I've read that it is something people do with Ingresses (https://github.com/cert-manager/cert-manager/issues/841#issuecomment-414299467) by pre-creating the Certificate resource instead of letting it be created by the ingress shim.

Enable the use of all fields in a Certificate even if there are not supported annotations

Ah, that's an excellent point. Ingresses and Gateways give you the opportunity to use a pre-existing Certificate resource, letting you fine-tune the Certificate, and among others letting you configure the encoding of the certificate.

I also understand for need for the new annotation cert-manager.io/certificate-name. For Ingress and Gateway resources, you just have to reference the Secret in the Ingress. But for Routes, something needs to copy the Secret into the Route's spec.key and spec.certificate. Which could be done by cert-utils-operator I guess.

If using a re-encrypt route along with the PEM format used in the Route.

Ingress and Gateway resources both let you re-encrypt traffic, so I do understand the need here. I can't see how people would be able to decrypt at the backend's site by reading the Route's spec.key and spec.certificate. Reading a Secret is much more convenient.

The option of supporting these features in cert-manager routes would still be possible via annotations the same way it is for cert-manager and ingress objects, but only as a means to configure the Certificate resource and any complex logic can be offloaded to cert-manager.

Just to understand things better, I compared the annotations supported by each:

csi-driver	Ingress, Gateway	Route
csi.cert-manager.io/issuer-name		cert-manager.io/issuer-name
	cert-manager.io/issuer	cert-manager.io/issuer
	cert-manager.io/cluster-issuer	cert-manager.io/cluster-issuer
csi.cert-manager.io/issuer-kind	cert-manager.io/issuer-kind	cert-manager.io/issuer-kind
csi.cert-manager.io/issuer-group	cert-manager.io/issuer-group	cert-manager.io/issuer-group
csi.cert-manager.io/common-name	cert-manager.io/common-name	cert-manager.io/common-name
csi.cert-manager.io/dns-names	?	cert-manager.io/alt-names
csi.cert-manager.io/ip-sans	cert-manager.io/ip-sans	cert-manager.io/ip-sans
csi.cert-manager.io/uri-sans	cert-manager.io/uri-sans	cert-manager.io/uri-sans
	cert-manager.io/email-sans	cert-manager.io/email-sans
csi.cert-manager.io/duration	cert-manager.io/duration	cert-manager.io/duration
csi.cert-manager.io/is-ca
csi.cert-manager.io/certificate-file
csi.cert-manager.io/ca-file
csi.cert-manager.io/privatekey-file
csi.cert-manager.io/fs-group
csi.cert-manager.io/renew-before	cert-manager.io/renew-before	cert-manager.io/renew-before
csi.cert-manager.io/pkcs12-enable
csi.cert-manager.io/pkcs12-filename
csi.cert-manager.io/pkcs12-password
	kubernetes.io/tls-acme
	acme.cert-manager.io/http01-ingress-class
	acme.cert-manager.io/http01-edit-in-place
	cert-manager.io/subject-organizations	cert-manager.io/subject-organizations
	cert-manager.io/subject-organizationalunits	cert-manager.io/subject-organizationalunits
	cert-manager.io/subject-countries	cert-manager.io/subject-countries
	cert-manager.io/subject-provinces	cert-manager.io/subject-provinces
	cert-manager.io/subject-localities	cert-manager.io/subject-localities
	cert-manager.io/subject-postalcodes	cert-manager.io/subject-postalcodes
	cert-manager.io/subject-streetaddresses	cert-manager.io/subject-streetaddresses
	cert-manager.io/subject-serialnumber	cert-manager.io/subject-serialnumber
csi.cert-manager.io/key-usages	cert-manager.io/usages
	cert-manager.io/revision-history-limit
	cert-manager.io/private-key-algorithm	cert-manager.io/private-key-algorithm
csi.cert-manager.io/key-encoding	cert-manager.io/private-key-encoding
	cert-manager.io/private-key-size	cert-manager.io/private-key-size
csi.cert-manager.io/reuse-private-key	cert-manager.io/private-key-rotation-policy

In terms of annotations, there are only missing:

• revision history limit,
• key usages.

[maelvls]

I think [the duplicate logic for handling the creation of certificates] is the case for multiple cert-manager projects: csi-driver, csi-driver-spiffe, istio-csr, ... (https://github.com/cert-manager/openshift-routes/pull/55#issuecomment-1972996170)

Routes are meant to behave like Ingress and Gateway resource, aren't they? Ingress and Gateway resources create Certificate resources, it would thus make sense that Routes create Certificates.

As Craig pointed out, Ingress and Gateway resources support re-encryption and sharing certificates across resources. For the sake of consistency, I'd suggest we go with Certificates so that the experience between Ingresses, Gateways, and Routes are similar.

[maelvls]

On 7 March 2024, during the dev biweekly meetings (meeting notes), Trilok Geer – from Red hat – talked about OpenShift 4.16 (unreleased) and the new field externalCertificate. This ties well with the idea of creating Certificates instead of CertificateRequests in openshift-routes. Right now, externalCertificate's API isn't clear to us (is it meant to target a Secret resource)? Trilok told me that he will get back to us soon.

On 16 May 2024, we talked a bit more on that topic (meeting notes):

• On the topic of bringing support for the Routes CRD to cert-manager "core", the status quo hasn't changed: although technically acceptable as long as it stays in a separate binary with its own go.mod, none of us have motivation implementing that. That said, there were concerns over the fact that only OpenShift 4.16 users would be able to benefit this.
• Red Hat needs to be more involved with these discussions. I'll try to get someone from Red Hat for the next dev biweekly meeting.
• There needs to be a discussion around how Red Hat contributes to cert-manager, in particular the openshift-routes project. Right now, openshift-routes doesn't have end-to-end tests because OpenShift costs a fortune to run.

[hawksight]

Moving from Certificates to CertificateRequests: A Survey

@maelvls is the title the wrong way round? And should be:

Moving from CertificateRequests to Certificates: A Survey

[TrilokGeer]

@maelvls , thanks for sharing the collective date on annotations. This gives a better insight about compatibility with configurations supported by annotations.

externalCertificate's API isn't clear to us (is it meant to target a Secret resource)?

Yes, the externalCertificate targets a secret resource. The secret is generated and managed by cert-manager via the usual flow of Certificate resource allowing integration with cert-manager. This approach overcomes the problem of private key in route configuration.

[mf-of]

Just to chime in here: I fully agree with @jacksgt here, for exactly the same reasons as he points out. We've seen certificates that weren't able to be renewed at all unless manual action was taken, which kind of defeats the purpose of cert-manager altogether (removing Order and CertificateRequest was required in all cases we've seen). I'm hoping that using Certificates instead of CertificateRequests will improve this.

We haven't been using it all that long yet, but the combination of cert-manager, openshift-routes addition, and Let's Encrypt has been problematic so far (curiously, no issues when Let's Encrypt is replaced with a local Step CA, with renewals every 30m).

[SgtCoDFish]

We've now merged #101 which makes the switch to using Certificates unilaterally. It'll be included in the next release which should be coming soon!

[Bengrunt]

We've now merged #101 which makes the switch to using Certificates unilaterally. It'll be included in the next release which should be coming soon!

Hello, thanks everyone for that piece of work! For us this will solve the issue of having concurrent certificate requests failing randomly in situations with multiple routes using the same domain but different subpaths.

Any insight on how transition will occur from the CertificateRequests to the Certificates mode ? Will existing requests be picked up and certificates moved to new Certificates ? I guess I'll have to try :)

Thanks !

[SgtCoDFish]

Any insight on how transition will occur from the CertificateRequests to the Certificates mode ? Will existing requests be picked up and certificates moved to new Certificates ? I guess I'll have to try :)

First of all: please do try and report back! If there's anything we need to tweak now's the time.

To answer your question: after upgrading, the same renewal logic as before will be used in that if a cert is found on the route and it's valid and doesn't need to be renewed, it'll be left in place.

When the cert is renewed, it'll transparently be renewed just with a Certificate rather than a CertificateRequest.

I triggered a renewal locally for testing by manually deleting the certificate from the Route, which immediately triggered a reissuance.

[jacksgt]

Thanks for your input and considerations everyone, I'm really happy to see this has been released. :-)