kcert
kcert copied to clipboard
nabsul
new nginx-ingress annotation
Hi,
as stated in your documentation kcert does expect following annotation:
metadata:
annotations:
kubernetes.io/ingress.class: "nginx"
but newer versions of nginx-ingress changed to:
spec:
ingressClassName: nginx
see also: https://kubernetes.github.io/ingress-nginx/user-guide/basic-usage/
Any consideration to support both ways since I wasn't able to generate tls certificate with the latest cert-manager.
Thx a lot
/david
Good timing, this is a change I can make in the next release that I'm preparing.
I think I'll simply do both: The annotation and the ingressClass, to cover both scenarios. I don't think it will hurt anything to have both.
Implemented in: https://github.com/nabsul/kcert/pull/60/commits/95b060f7a49eb49b74adfff5fd74d3cd94ba7a7a
Wow, that was quick, thx a lot.
Don't want to stress you, but when did you plan to publish the new release? Just to get an idea.
As soon at its out, will give it again a try.
Thx
/david
This weekend the latest. I'll close this issue when I do so you get notified 🙂
@knottnhupfer I haven't got around to publishing an official release yet, but the ingressClass fix is merged into main and you should be able to use it with the nabsul/kcert:sha-aafd15f tag.
Ohh, great news and thx for your effort! Will give it a try latest next week and come back to you.
Thx
Finally I was able to test it with the release, was not able to deploy the tag ...
Kubernetes Version: 1.24.6
I always get an error when creating the ingress on azure kubernetes:
I1021 11:14:59.570517 1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"kcert-test", UID:"dbffcf9c-423e-4a02-b5d2-175c2569e22f", APIVersion:"networking.k8s.io/v1", ResourceVersion:"22800", FieldPath:""}): type: 'Warning' reason: 'AddedOrUpdatedWithWarning' Configuration for default/kcert-test was added or updated ; with warning(s): TLS secret kcert-test-tls is invalid: secret doesn't exist or of an unsupported type
Here a link to kcert log file related to this ingress: https://gist.github.com/knottnhupfer/f822650315535a19e3c14f769ecc1cb8
I expected the secret to be created automatically. My Ingress configuration looks like:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: kcert-test
labels:
kcert.dev/ingress: "managed"
spec:
ingressClassName: nginx
tls:
- secretName: kcert-test-tls
hosts:
- kcert-test.northeurope.cloudapp.azure.com
rules:
- host: kcert-test.northeurope.cloudapp.azure.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: http-server-service
port:
number: 80
Hi, are you able to see any of the logs from KCert itself?
The first thing I would check is that KCert is accessible from the outside. If you create in ingress to KCert's port 80, you should be able to access http://kcert-test.northeurope.cloudapp.azure.com/.well-known/acme-challenge/test/hello
Can you also check if KCert has created an ingress object for itself? This should be in the kcert namespace by default.
Hi,
guess you spotted the right place. Seems like when creating the kcert ingress there are some problems with the annotations, if I understood the error message right.
here the logs from kcert pod after reinstalling kcert and enabled tls afterwards:
{
"Timestamp": "2022:10:22 07:06:08.879259",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.IngressMonitorService",
"Message": "Ingress change event [Added] for default-kcert-test",
"State": {
"Message": "Ingress change event [Added] for default-kcert-test",
"type": "Added",
"ns": "default",
"name": "kcert-test",
"{OriginalFormat}": "Ingress change event [{type}] for {ns}-{name}"
}
}
{
"Timestamp": "2022:10:22 07:06:08.981495",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.IngressMonitorService",
"Message": "Processing ingress default:kcert-test",
"State": {
"Message": "Processing ingress default:kcert-test",
"ns": "default",
"n": "kcert-test",
"{OriginalFormat}": "Processing ingress {ns}:{n}"
}
}
{
"Timestamp": "2022:10:22 07:06:08.981756",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.IngressMonitorService",
"Message": "Processing secret kcert-test-tls",
"State": {
"Message": "Processing secret kcert-test-tls",
"s": "kcert-test-tls",
"{OriginalFormat}": "Processing secret {s}"
}
}
{
"Timestamp": "2022:10:22 07:06:08.983595",
"EventId": 0,
"LogLevel": "Information",
"Category": "KCert.Services.IngressMonitorService",
"Message": "Handling cert default - kcert-test-tls hosts: kcert-test.northeurope.cloudapp.azure.com",
"State": {
"Message": "Handling cert default - kcert-test-tls hosts: kcert-test.northeurope.cloudapp.azure.com",
"ns": "default",
"name": "kcert-test-tls",
"h": "kcert-test.northeurope.cloudapp.azure.com",
"{OriginalFormat}": "Handling cert {ns} - {name} hosts: {h}"
}
}
{
"Timestamp": "2022:10:22 07:06:09.033293",
"EventId": 0,
"LogLevel": "Error",
"Category": "KCert.Services.KCertClient",
"Message": "HTTP Operation failed with respones: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Ingress.extensions \\"kcert\\" is invalid: annotations.kubernetes.io/ingress.class: Invalid value: \\"nginx\\": can not be set when the class field is also set","reason":"Invalid","details":{"name":"kcert","group":"extensions","kind":"Ingress","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \\"nginx\\": can not be set when the class field is also set","field":"annotations.kubernetes.io/ingress.class"}]},"code":422}\n",
"Exception": "k8s.Autorest.HttpOperationException: Operation returned an invalid status code 'UnprocessableEntity'\n at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)\n at k8s.AbstractKubernetes.k8s.INetworkingV1Operations.CreateNamespacedIngressWithHttpMessagesAsync(V1Ingress body, String namespaceParameter, String dryRun, String fieldManager, String fieldValidation, Nullable\u00601 pretty, IReadOnlyDictionary\u00602 customHeaders, CancellationToken cancellationToken)\n at k8s.NetworkingV1OperationsExtensions.CreateNamespacedIngressAsync(INetworkingV1Operations operations, V1Ingress body, String namespaceParameter, String dryRun, String fieldManager, String fieldValidation, Nullable\u00601 pretty, CancellationToken cancellationToken)\n at KCert.Services.K8sClient.CreateIngressAsync(V1Ingress ingress) in /build/Services/K8sClient.cs:line 173\n at KCert.Services.KCertClient.AddChallengeHostsAsync(IEnumerable\u00601 hosts) in /build/Services/KCertClient.cs:line 107\n at KCert.Services.KCertClient.RenewCertAsync(Task prev, String ns, String secretName, String[] hosts, CancellationToken tok) in /build/Services/KCertClient.cs:line 61",
"State": {
"Message": "HTTP Operation failed with respones: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Ingress.extensions \\"kcert\\" is invalid: annotations.kubernetes.io/ingress.class: Invalid value: \\"nginx\\": can not be set when the class field is also set","reason":"Invalid","details":{"name":"kcert","group":"extensions","kind":"Ingress","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \\"nginx\\": can not be set when the class field is also set","field":"annotations.kubernetes.io/ingress.class"}]},"code":422}\n",
"resp": "{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Ingress.extensions \\"kcert\\" is invalid: annotations.kubernetes.io/ingress.class: Invalid value: \\"nginx\\": can not be set when the class field is also set","reason":"Invalid","details":{"name":"kcert","group":"extensions","kind":"Ingress","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \\"nginx\\": can not be set when the class field is also set","field":"annotations.kubernetes.io/ingress.class"}]},"code":422}\n",
"{OriginalFormat}": "HTTP Operation failed with respones: {resp}"
}
}
{
"Timestamp": "2022:10:22 07:06:09.060249",
"EventId": 0,
"LogLevel": "Error",
"Category": "KCert.Services.IngressMonitorService",
"Message": "Ingress watch event handler failed unexpectedly",
"Exception": "k8s.Autorest.HttpOperationException: Operation returned an invalid status code 'UnprocessableEntity'\n at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)\n at k8s.AbstractKubernetes.k8s.INetworkingV1Operations.CreateNamespacedIngressWithHttpMessagesAsync(V1Ingress body, String namespaceParameter, String dryRun, String fieldManager, String fieldValidation, Nullable\u00601 pretty, IReadOnlyDictionary\u00602 customHeaders, CancellationToken cancellationToken)\n at k8s.NetworkingV1OperationsExtensions.CreateNamespacedIngressAsync(INetworkingV1Operations operations, V1Ingress body, String namespaceParameter, String dryRun, String fieldManager, String fieldValidation, Nullable\u00601 pretty, CancellationToken cancellationToken)\n at KCert.Services.K8sClient.CreateIngressAsync(V1Ingress ingress) in /build/Services/K8sClient.cs:line 173\n at KCert.Services.KCertClient.AddChallengeHostsAsync(IEnumerable\u00601 hosts) in /build/Services/KCertClient.cs:line 107\n at KCert.Services.KCertClient.RenewCertAsync(Task prev, String ns, String secretName, String[] hosts, CancellationToken tok) in /build/Services/KCertClient.cs:line 61\n at KCert.Services.IngressMonitorService.TryUpdateSecretAsync(String ns, String name, IEnumerable\u00601 hosts, CancellationToken tok) in /build/Services/IngressMonitorService.cs:line 124\n at KCert.Services.IngressMonitorService.HandleIngressEventAsync(WatchEventType type, V1Ingress ingress, CancellationToken tok) in /build/Services/IngressMonitorService.cs:line 94",
"State": {
"Message": "Ingress watch event handler failed unexpectedly",
"{OriginalFormat}": "Ingress watch event handler failed unexpectedly"
}
}
Thanks for pulling these logs! I think the key is this message:
"Message": "HTTP Operation failed with respones: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Ingress.extensions \\"kcert\\" is invalid: annotations.kubernetes.io/ingress.class: Invalid value: \\"nginx\\": can not be set when the class field is also set","reason":"Invalid","details":{"name":"kcert","group":"extensions","kind":"Ingress","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \\"nginx\\": can not be set when the class field is also set","field":"annotations.kubernetes.io/ingress.class"}]},"code":422}\n",
It appears that in Azure you can't have both the annotation and the ingressClass at the same time. I wonder why? This doesn't seem to be harmful, and I thought I was being safe by doing both :-) This was not an issue for me when testing in Digital Ocean.
Anyways, the fix should be easy. I'll make it default to using ingressClass, with the option to switch back to the annotation.
