openshift-routes
openshift-routes copied to clipboard
cert-manager
Route with cert-manager annotations is not created
Hi, I'm trying to install a route of edge termination using cert-manager annotations in openshift cluster (crc), this is part of our operator's reconcile loop. The following annotations are added to the route:
"annotations": {
"cert-manager.io/common-name": "arkamq.io",
"cert-manager.io/issuer-kind": "Issuer",
"cert-manager.io/issuer-name": "selfsigned-issuer-ingress"
}
The dump of the full Route spec:
"route": {
"kind": "Route",
"apiVersion": "v1",
"metadata": {
"name": "artemis-broker-wconsj-0-svc-rte",
"namespace": "default",
"creationTimestamp": null,
"labels": {
"ActiveMQArtemis": "artemis-broker",
"application": "artemis-broker-app",
"statefulset.kubernetes.io/pod-name": "artemis-broker-ss-0"
},
"annotations": {
"cert-manager.io/common-name": "arkamq.io",
"cert-manager.io/issuer-kind": "Issuer",
"cert-manager.io/issuer-name": "selfsigned-issuer-ingress"
}
},
"spec": {
"host": "artemis-broker-wconsj-0-svc-rte-default.apps-crc.testing",
"to": {
"kind": "Service",
"name": "artemis-broker-wconsj-0-svc",
"weight": null
},
"port": {
"targetPort": "wconsj-0"
},
"tls": {
"termination": "edge",
"insecureEdgeTerminationPolicy": "Redirect"
},
"wildcardPolicy": "None"
},
"status": {
"ingress": null
}
}
However the route is not created. I dumped the openshift-route pod's log and I see something is going on but no obvious error messages. I'll attach the log. My environment is
CRC version: 2.25.0+97705a
OpenShift version: 4.13.6
Podman version: 4.4.4
cert-manager is installed from operatorHub from default channel
openshit-route is installed using the following command
oc apply -f https://github.com/cert-manager/openshift-routes/releases/latest/download/cert-manager-openshift-routes.yaml
I'll attach also the output of oc events to provide more information.
Please help. Thanks!
oc_events.log
openshift-routes-pod.log
It looks like there is an issue with the route definition? In the logs the route is visible, but in the pasted configuration apiVersion: v1 for kind Route is not valid, it should be apiVersion: route.openshift.io/v1. This might just be a copy paste issue though?
Can you also check to see whether a CertificateRequest CR was generated? I don't see the event for it in your attachment but there is a log message the CR is not ready. If it was cceated, can you check to make sure the issuer is in a Ready state in the default namespace and perhaps check your cert-manager controller logs (assuming the certificate request was created).
$ oc get issuers -n default
$ oc get certificaterequests -n default
$ oc describe certificaterequest <cr>
$ oc get events -n cert-manager
$ oc get events -n default
$ oc logs cert-manager-controller -n cert-manager
"CR not ready"
I0905 13:27:30.681290 1 app.go:90] cert-manager-openshift-routes/controller-manager "msg"="Event(v1.ObjectReference{Kind:\"Route\", Namespace:\"default\", Name:\"artemis-broker-wconsj-0-svc-rte\", UID:\"345b4999-f3db-4dc4-bc4c-2c842fb9f4f1\", APIVersion:\"route.openshift.io/v1\", ResourceVersion:\"140254\", FieldPath:\"\"}): type: 'Normal' reason: 'Issuing' Issuing cert as no certificate exists"
I0905 13:27:30.690862 1 sync.go:99] cert-manager-openshift-routes/route/sync "msg"="cr is not ready yet" "resourceVersion"="140254" "route"={"Namespace":"default","Name":"artemis-broker-wconsj-0-svc-rte"}
I was able to successfully generate a certificate using your provided Route spec, aside from the apiVersion that needed a correction.
@ctrought Hi,
I tried to fix the route's apiVersion (it's in our operator code) and re-deployed our resources. I can see the apiVersion is correct now (route.openshift.io/v1).
But the route is still not able to be deployed. The route is created/deployed by our operator based on
our deployed CR. I observed during the process the route is first being created and then it's gone.
(I'm using oc get routes command)
Here is the output of the command I tried:
- first I tried to get route:(i have everything in default namespace)
$ oc get route
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
artemis-broker-wconsj-0-svc-rte artemis-broker-wconsj-0-svc-rte-default.apps-crc.testing artemis-broker-wconsj-0-svc wconsj-0 edge/Redirect None
- A few seconds later the route is gone:
$ oc get route
No resources found in default namespace.
- get issuer
$ oc get issuers -n default
NAME READY AGE
selfsigned-issuer-ingress True 2m19s
- get csr
$ oc get certificaterequests -n default
No resources found in default namespace.
- get cert-manager events
$ oc get events -n cert-manager (see the attached oc_get_events_n_cert_manager.output.log)
- get default events
$ oc get events -n default (see attached oc_get_events_n_default.output.log )
- get cert-manager log
$ oc logs cert-manager-7cc7959fbb-w6fzk -n cert-manager (see attached oc_logs_cert_manager.log)
- get openshift-routes log
$ oc logs cert-manager-openshift-routes-5bfbc6fb88-75zvd -n cert-manager (see attached cert-manager-openshift-routes-pod.log)
In oc_get_events_n_default.output I see some error like
7m20s Warning InternalReconcileError route/artemis-broker-wconsj-0-svc-rte error while reconciling: Operation cannot be fulfilled on routes.route.openshift.io "artemis-broker-wconsj-0-svc-rte": the object has been modified; please apply your changes to the latest version and try again
Not sure if that's relevant. But I do see the CSR get approved in the same file:
m31s Normal cert-manager.io certificaterequest/artemis-broker-wconsj-0-svc-rte-q4r49 Certificate request has been approved by cert-manager.io
I'm not sure what could be the issue. Please help.
Thanks cert-manager-openshift-routes-pod.log oc_get_events_n_cert_manager.output.log oc_get_events_n_default.output.log oc_logs_cert_manager.log
I observed during the process the route is first being created and then it's gone
That sounds like something else is causing it. Cert-manager routes won't delete your route. Maybe whatever you're using to deploy the route is removing the route?
Have you tried deploying the route directly instead of using the operator you mentioned? You can check the kube api-server audit logs to see what's deleting the route. Since you're using OpenShift, you can use the cluster logging operator to send audit logs to elasticsearch or loki to query the audit logs if you don't already. Or you can use the CLI to view them https://docs.openshift.com/container-platform/4.13/security/audit-log-view.html
Thanks. Yes turns out it's out operator's bug that causes the Route resource being deleted.
Now with the bug fixed the route seems working (I can access the service via the route). However it looks like the route is not using the certificate from openshift-route. There is no cert resource generated (in default namespace). By examining the certificate it seems that the route is using the default certificate.
The route spec does show some fields being generated by openshift-route, but it's not complete.
For example if I deploy this route:
apiVersion: route.openshift.io/v1
kind: Route
metadata:
annotations:
cert-manager.io/common-name: arkamq.io
cert-manager.io/issuer-kind: Issuer
cert-manager.io/issuer-name: selfsigned-issuer-ingress
labels:
ActiveMQArtemis: artemis-broker
application: artemis-broker-app
statefulset.kubernetes.io/pod-name: artemis-broker-ss-0
name: my-route-artemis-broker
namespace: default
spec:
host: my-route-artemis-broker-default.apps-crc.testing
port:
targetPort: wconsj-0
tls:
insecureEdgeTerminationPolicy: Redirect
termination: edge
to:
kind: Service
name: artemis-broker-wconsj-0-svc
weight: 100
and then use oc get route to get the created route, there are some new annotations
injected (cert-manager.io/certificate-revision and cert-manager.io/next-private-key)
But no certs in the route spec. See below for the full contents:
(20)[20:51:26]@route$ oc get route my-route-artemis-broker -o yaml
apiVersion: route.openshift.io/v1
kind: Route
metadata:
annotations:
cert-manager.io/certificate-revision: "0"
cert-manager.io/common-name: arkamq.io
cert-manager.io/issuer-kind: Issuer
cert-manager.io/issuer-name: selfsigned-issuer-ingress
cert-manager.io/next-private-key: |
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAudmUOn8c6KwCKB+ZI4tKBcUB1iJI3Baeh7gOpTcZmOmYJ3cn
Zq+H1lQ5xcdc9c4tkEEcLscNOlWZiCTpMmKZFXbPJxr8K4g7s/KasTBY8XtnkPTL
VQedaMiRy1aIJKYxE88ImUh/L/bY3iQIkUA4RedeaaTEE/CTFe+8+pOmTyEH/Ta4
atNAffGbnhwHKOxjuMW3n0ePvHP0twSFxxRA+n6wHHzREftcDmyo3rr1MGfMwwh0
7jAphdZ3jFrz+9glr6PUwdiK14bAYiQRDr99PabLtRqtADpXBuKmEr3RIoJ0R10Q
YC+LFJ8l0jsp4yXdOkPAMTN8c9iLkesbm10IfwIDAQABAoIBAHbXBazeKWsyynJD
ulX7NIs57bGy3Ko3oImijWLiN/EKtivOSpij95V66DJrCCt5+rK6n73dtLyvjex7
xc5dNAyAm3gjDkr740lvAbH5+3X29kO/9OFbbDFbx7ahj4734nwLMDFsJUBTdVue
6QeQbCtvBnVBUcx6UmN9mHcWwQrpOIs/5SpI49TNixN641k6JxE9YG060Ih1kEMg
x0rE6R3Spb09/aFcFvJLgloJD4GC3IZEkZoqFvBc4myg7t3JuuA8HXfE34mcHLff
NaAIjlVrO14e83NOMddIvrLqMXybJC3DUST+Cl9ZwMCwiWSteTsjkTv+b3aDnWrB
/vATodkCgYEA9B8mWahrrFHe3XVh10qmbjlcGil4FU9T8dy4M9VKPEq8OOUWjvEe
dWt7lX97VHy6jCjInJObLsbzZxiacea6E6t78V2QxvcCAR9uDjN4NDzKB/xIs9we
U7rargeohUZNd8fncYEKO5TCzNN97k4+uzzC8i0pvI7ja1CSMVn+5gUCgYEAwuSU
TgIQTutBxmGf4jmjlhH+6lK1QTdM1TW9bvtyI2i/JrvuRU8T0RzrMfFvU+lER3Su
D20M9goFDgL51GsKKfmbJOWO22lAzz3ASHbl6+M7POY8pzb/+QZ2AAX+yHfOO2sz
I7RxwN6gD0Jz/PcrVOJ9zZWw/hgX8LVeWo1k17MCgYAb9Z1c029RkyblUlZcZA5f
vW4RCQLZ3fYNU2w9c4ZHFYAtO/+za2ZDqIXE/47OhGKRPvKLDFTPrLWf7y9xMC3L
pu5RHJhCUXQHUOQ3+CkiQnwZIgnpn6JRkKsPF41zhjPFLj4Rv+e+HzGYZ/iaXJfO
tgzV09pd4DenyNiVkmshSQKBgF3Yv+lIaR33fvbOL7U52CH9jGi9ZvZ+cIsCk7OL
dQzD9bfrKvNI/tvlNwQXpGodm3llICPheSpI7OIwYKj+2igYP7Lmr4HMDx88Nwkt
581M4EuU+oPyohQCHnQobuWNe0t1dWyuCVC8Ywq2DzAyvTTk03tZp57Tb0PDsimc
iICtAoGBAOOZeVUpkHY/oDsMyenTfJWDxKTtncnNlV0tG5QII9J/AAdRX7gNGIrq
cp1ubde4LZVpvB5HXf5LF+12ZZDbFCbDTK6Kx8HoDLRcv/1vs/A/heXdnxl1e4eh
vRtYoO8fDx3tVC4QSdWoJr2HjNXp5GO28rviJusafeNfhAaEirwl
-----END RSA PRIVATE KEY-----
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"route.openshift.io/v1","kind":"Route","metadata":{"annotations":{"cert-manager.io/common-name":"arkamq.io","cert-manager.io/issuer-kind":"Issuer","cert-manager.io/issuer-name":"selfsigned-issuer-ingress"},"labels":{"ActiveMQArtemis":"artemis-broker","application":"artemis-broker-app","statefulset.kubernetes.io/pod-name":"artemis-broker-ss-0"},"name":"my-route-artemis-broker","namespace":"default"},"spec":{"host":"my-route-artemis-broker-default.apps-crc.testing","port":{"targetPort":"wconsj-0"},"tls":{"insecureEdgeTerminationPolicy":"Redirect","termination":"edge"},"to":{"kind":"Service","name":"artemis-broker-wconsj-0-svc","weight":100}}}
creationTimestamp: "2023-09-19T09:20:20Z"
labels:
ActiveMQArtemis: artemis-broker
application: artemis-broker-app
statefulset.kubernetes.io/pod-name: artemis-broker-ss-0
name: my-route-artemis-broker
namespace: default
resourceVersion: "1060469"
uid: 1d98e663-18b5-4ce8-bf1c-b8521068370f
spec:
host: my-route-artemis-broker-default.apps-crc.testing
port:
targetPort: wconsj-0
tls:
insecureEdgeTerminationPolicy: Redirect
termination: edge
to:
kind: Service
name: artemis-broker-wconsj-0-svc
weight: 100
wildcardPolicy: None
status:
ingress:
- conditions:
- lastTransitionTime: "2023-09-19T09:20:20Z"
status: "True"
type: Admitted
host: my-route-artemis-broker-default.apps-crc.testing
routerCanonicalHostname: router-default.apps-crc.testing
routerName: default
wildcardPolicy: None
I have no idea what's the problem. Please help. If you need more info please let me know. Thanks
@ctml91 and I check the audit log and there seems to be no specific info regarding the route. Everything is pretty general.
Can you gather the logs again from the controller, etc.?
$ oc get issuers -n default
$ oc get certificaterequests -n default
$ oc describe certificaterequest <cr>
$ oc get events -n default
$ oc get events -n cert-manager
$ oc logs <cert-manager-route-controller> -n cert-manager
$ oc logs <cert-manager-controller> -n cert-manager
Hi @ctrought Here are the information I gathered.
$ oc get issuers -n default
(20)[23:22:08]@route$ oc get issuers -n default
NAME READY AGE
selfsigned-issuer-ingress True 11h
$ oc get certificaterequests -n default
(20)[23:22:17]@route$ oc get certificaterequests -n default
NAME APPROVED DENIED READY ISSUER REQUESTOR AGE
artemis-broker-wconsj-0-svc-rte-s5jd4 True False selfsigned-issuer-ingress system:serviceaccount:cert-manager:cert-manager-openshift-routes 11h
my-route-artemis-broker-njg94 True False selfsigned-issuer-ingress system:serviceaccount:cert-manager:cert-manager-openshift-routes 6h2m
$ oc describe certificaterequest
(20)[23:23:22]@route$ oc describe certificaterequests my-route-artemis-broker-njg94
Name: my-route-artemis-broker-njg94
Namespace: default
Labels: <none>
Annotations: cert-manager.io/certificate-revision: 1
API Version: cert-manager.io/v1
Kind: CertificateRequest
Metadata:
Creation Timestamp: 2023-09-19T09:20:20Z
Generate Name: my-route-artemis-broker-
Generation: 1
Managed Fields:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:cert-manager.io/certificate-revision:
f:generateName:
f:ownerReferences:
.:
k:{"uid":"1d98e663-18b5-4ce8-bf1c-b8521068370f"}:
f:spec:
.:
f:duration:
f:issuerRef:
.:
f:kind:
f:name:
f:request:
f:usages:
Manager: Go-http-client
Operation: Update
Time: 2023-09-19T09:20:20Z
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:conditions:
.:
k:{"type":"Approved"}:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
Manager: cert-manager-certificaterequests-approver
Operation: Update
Subresource: status
Time: 2023-09-19T09:20:20Z
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:conditions:
k:{"type":"Ready"}:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
f:failureTime:
Manager: cert-manager-certificaterequests-issuer-selfsigned
Operation: Update
Subresource: status
Time: 2023-09-19T09:20:20Z
Owner References:
API Version: route.openshift.io/v1
Block Owner Deletion: true
Controller: true
Kind: Route
Name: my-route-artemis-broker
UID: 1d98e663-18b5-4ce8-bf1c-b8521068370f
Resource Version: 1060483
UID: 13169b94-65cc-4c79-a80f-6b09449c996a
Spec:
Duration: 2160h0m0s
Extra:
authentication.kubernetes.io/pod-name:
cert-manager-openshift-routes-5bfbc6fb88-75zvd
authentication.kubernetes.io/pod-uid:
7f3e6354-93c1-4911-9bda-be692822a411
Groups:
system:serviceaccounts
system:serviceaccounts:cert-manager
system:authenticated
Issuer Ref:
Kind: Issuer
Name: selfsigned-issuer-ingress
Request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ3B6Q0NBWThDQVFBd0ZERVNNQkFHQTFVRUF4TUpZWEpyWVcxeExtbHZNSUlCSWpBTkJna3Foa2lHOXcwQgpBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF1ZG1VT244YzZLd0NLQitaSTR0S0JjVUIxaUpJM0JhZWg3Z09wVGNaCm1PbVlKM2NuWnErSDFsUTV4Y2RjOWM0dGtFRWNMc2NOT2xXWmlDVHBNbUtaRlhiUEp4cjhLNGc3cy9LYXNUQlkKOFh0bmtQVExWUWVkYU1pUnkxYUlKS1l4RTg4SW1VaC9ML2JZM2lRSWtVQTRSZWRlYWFURUUvQ1RGZSs4K3BPbQpUeUVIL1RhNGF0TkFmZkdibmh3SEtPeGp1TVczbjBlUHZIUDB0d1NGeHhSQStuNndISHpSRWZ0Y0RteW8zcnIxCk1HZk13d2gwN2pBcGhkWjNqRnJ6KzlnbHI2UFV3ZGlLMTRiQVlpUVJEcjk5UGFiTHRScXRBRHBYQnVLbUVyM1IKSW9KMFIxMFFZQytMRko4bDBqc3A0eVhkT2tQQU1UTjhjOWlMa2VzYm0xMElmd0lEQVFBQm9FNHdUQVlKS29aSQpodmNOQVFrT01UOHdQVEE3QmdOVkhSRUVOREF5Z2pCdGVTMXliM1YwWlMxaGNuUmxiV2x6TFdKeWIydGxjaTFrClpXWmhkV3gwTG1Gd2NITXRZM0pqTG5SbGMzUnBibWN3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUY1WWxGN3AKck1Pc0RyNGd3SWw4eVJkY1N2bGQ3VzFzaGxqMTVXRkwwd2JMWXRwSG5FRE5Lb1ZrS0xXRmNlRlQyU25wb2JjMgp6UnFySUFZU3oxeThZejBGTHJLQ1EwVG9tbXFMQXNsc0RPZUt6ZGhXOWJnYkFOcW5IUmowajUwb2I2WjJ1dCtyCkJ6TENSWHVpY3pDaUFPUjNHVVFkL2J6VGtFU2ZyUFoybStjZlF3THZnUEUrc2gzWFBaSHpwNVlyRDhlenpLY24KVHpBREJHcXRRL1puc1liWkMxQUVrMEJBS1JuOEppb05ybXAxRTVFandtclB5eldIOTlSdW9vV0VTS2tsRExOVQpHVHRyT3lxb3Q1enUwSmlqYU5YeWswby9BZGxHaDF0QStuckNuZDhSLzBJT1NvUlcrMU9Ca0tOelFZTUp4bmNoClYxREFJZFNKTE1aY2hjRT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==
UID: 0d32d56f-00fe-4f7c-affe-f1d01a5c6598
Usages:
server auth
digital signature
key encipherment
Username: system:serviceaccount:cert-manager:cert-manager-openshift-routes
Status:
Conditions:
Last Transition Time: 2023-09-19T09:20:20Z
Message: Certificate request has been approved by cert-manager.io
Reason: cert-manager.io
Status: True
Type: Approved
Last Transition Time: 2023-09-19T09:20:20Z
Message: Annotation "cert-manager.io/private-key-secret-name" missing or reference empty: secret name missing
Reason: Failed
Status: False
Type: Ready
Failure Time: 2023-09-19T09:20:20Z
Events: <none>
$ oc get events -n default
(20)[23:28:34]@route$ oc get events -n default
No resources found in default namespace.
$ oc get events -n cert-manager
(20)[23:29:09]@route$ oc get events -n cert-manager
No resources found in cert-manager namespace.
$ oc logs
oc logs cert-manager-openshift-routes-5bfbc6fb88-75zvd -n cert-manager > cert-manager-openshift-routes.log
$ oc logs
oc logs cert-manager-7cc7959fbb-w6fzk -n cert-manager > cert-manager.log
cert-manager.log cert-manager-openshift-routes.log
Annotation "cert-manager.io/private-key-secret-name" missing or reference empty: secret name missing
I'm not sure what this annotation is for?
I just realized that openshift-routes is not compatible with cert-manager's SelfSigned issuer at the moment. The SelfSigned issuer requires the private key to be stored in a Secret resource and requires the special annotation cert-manager.io/private-key-secret-name to be present on the CertificateRequest.
Usually, cert-manager creates this annotation when it creates the CertificateRequest. But since openshift-routes is the one that creates the CertificateRequest and openshift-routes doesn't know that the issuer is the SelfSigned issuer, it doesn't add the annotation and doesn't create the Secret.
I think we need to mention this limitation in the README! 😅
