nifikop
nifikop copied to clipboard
Published
20 hours ago •
konpyutaika
konpyutaika
secure cluster with cert-manager Letsencrypt issuer: controller certificate has invalid Common Name value
Bug Report
What did you do?
- Installed nifikop operator using helm chart (single namespace). - verified pod running
Also tried with no
namespaces, but ran into permission errors on startup due to missing role bindings.
NAMESPACE="nifi"
existing=$(kubectl get namespace ${NAMESPACE})
[ "${existing}" = '' ] && kubectl create namespace ${NAMESPACE}
# You have to create the namespace before executing following command
HELM_EXPERIMENTAL_OCI=1 helm install nifikop \
oci://ghcr.io/konpyutaika/helm-charts/nifikop \
--namespace=${NAMESPACE} \
--version 0.10.0 \
--set image.tag=v0.10.0-release \
--set resources.requests.memory=256Mi \
--set resources.requests.cpu=250m \
--set resources.limits.memory=256Mi \
--set resources.limits.cpu=250m \
--set certManager.clusterScoped=true \
--set namespaces={"${NAMESPACE}"}
helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
nifikop nifi 1 2022-03-27 17:53:38.599003529 +0000 UTC deployed nifikop-0.10.0 0.10.0-release
-
Applied cert-manager operator using manifests at https://github.com/jetstack/cert-manager/releases/download/v1.7.2/cert-manager.yaml
-
Created cloudflare-apitoken-secret using this script
cloudflareSecretName="cloudflare-apitoken-secret";
existingApiToken=$(kubectl -n cert-manager get secret ${cloudflareSecretName} --template='{{.data.apitoken}}' | base64 -d )
if [ "$?" = '0' ]; then
echo "Secret already exists for $cloudflareSecretName with value $existingApiToken";
fi
read -p "Enter the new https://dash.cloudflare.com/profile/api-tokens API Tokens (or nothing to skip/leave as is $existingApiToken): " secret
if [ $secret != "" ]; then
kubectl -n cert-manager delete secret $cloudflareSecretName
kubectl create secret generic -n cert-manager $cloudflareSecretName --from-literal=apitoken=$secret
fi
secret=${secret:-$existingApiToken}
echo "secret is:$secret"
result=$(curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
-H "Authorization: Bearer $secret" \
-H "Content-Type:application/json")
echo $result
- Applied letsencrypt staging issuer (catch all/no dns filter) - verified certificates were issued
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
# disableAccountKeyGeneration: true
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: [email protected]
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource that will be used to store the account's private key.
name: letsencrypt-staging-account-private-key
solvers:
- dns01:
cloudflare:
email: [email protected]
apiTokenSecretRef:
name: cloudflare-apitoken-secret
key: apitoken
- Installed zookeeper - verified pods running
# The bitnami/zookeeper chart templates create
# fully qualified ZOO_SERVERS hostnames using the
# namespace value.
# eg.
# - name: ZOO_SERVERS
# value: zookeeper-0.zookeeper-headless.zookeeper.svc.cluster.local:2888:3888::1 zookeeper-1.zookeeper-headless.zookeeper.svc.cluster.local:2888:3888::2 zookeeper-2.zookeeper-headless.zookeeper.svc.cluster.local:2888:3888::3
helm template zookeeper bitnami/zookeeper \
--set resources.requests.memory=256Mi \
--set resources.requests.cpu=250m \
--set resources.limits.memory=256Mi \
--set resources.limits.cpu=250m \
--set networkPolicy.enabled=true \
--set replicaCount=3 \
--namespace=${NAMESPACE:-tbdNamespace} \
> .target/manifests.yaml
- Applied NifiCluster manifest
apiVersion: nifi.konpyutaika.com/v1alpha1
kind: NifiCluster
metadata:
name: tlsnifi
spec:
service:
headlessEnabled: true
zkAddress: "zookeeper-svc.zookeeper:2181"
zkPath: "/tlsnifi"
clusterImage: "apache/nifi:1.12.1"
oneNifiNodePerNode: true
managedAdminUsers:
- identity : "[email protected]"
name: "nifiadmin"
managedReaderUsers:
- identity : "[email protected]"
name: "nifireader"
propagateLabels: true
nifiClusterTaskSpec:
retryDurationMinutes: 10
nodeConfigGroups:
default_group:
isNode: true
storageConfigs:
- mountPath: "/opt/nifi/nifi-current/logs"
name: logs
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/data"
name: data
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/flowfile_repository"
name: flowfile-repository
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/nifi-current/conf"
name: conf
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/content_repository"
name: content-repository
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
- mountPath: "/opt/nifi/provenance_repository"
name: provenance-repository
pvcSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
serviceAccountName: "default"
resourcesRequirements:
limits:
cpu: "2"
memory: 3Gi
requests:
cpu: "1"
memory: 1Gi
nodes:
- id: 0
nodeConfigGroup: "default_group"
- id: 1
nodeConfigGroup: "default_group"
listenersConfig:
clusterDomain: c2.bugslifesolutions.com
useExternalDNS: false
internalListeners:
- type: "https"
name: "https"
containerPort: 8443
- type: "cluster"
name: "cluster"
containerPort: 6007
- type: "s2s"
name: "s2s"
containerPort: 10000
sslSecrets:
tlsSecretName: "tlsnifi.c2.bugslifesolutions.com"
create: true
issuerRef:
group: cert-manager.io
kind: ClusterIssuer
name: letsencrypt-staging
externalServices:
- name: "nifi-cluster"
spec:
type: LoadBalancer
portConfigs:
- port: 8443
internalListenerName: "https"
readOnlyConfig:
# NifiProperties configuration that will be applied to the node.
nifiProperties:
webProxyHosts:
- nifi.cluster2.admin.bugslifesolutions.com
- nifi.c2.admin.bugslifesolutions.com
# Additionnals nifi.properties configuration that will override the one produced based
# on template and configurations.
overrideConfigs: |
nifi.ui.banner.text=NiFi
nifi.security.user.oidc.discovery.url=https://accounts.google.com/.well-known/openid-configuration
nifi.security.user.oidc.client.id=930711295780-i72an91pqj7rib88r23qfv5q7mth8hgv.apps.googleusercontent.com
nifi.security.user.oidc.client.secret=BvmEyr81P0YXZtIt1FIfGsRs
nifi.security.identity.mapping.pattern.dn=CN=([^,]*)(?:, (?:O|OU)=.*)?
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=NONE
What did you expect to see? A pod created for the NifiCluster.
What did you see instead? Under which circumstances?
No pods were created for the NifiCluster and the status.state is ClusterReconciling:
status:
nodesState:
"0":
configurationState: ""
gracefulActionState:
actionState: ""
errorMessage: ""
initClusterNode: true
podIsReady: false
prometheusReportingTask:
id: ""
version: 0
rollingUpgradeStatus:
errorCount: 0
lastSuccess: ""
state: ClusterReconciling
The Nifikop operator logs suggest the secrets aren't created appropriately:
{"level":"info","ts":1648403623.7630396,"logger":"setup","msg":"manager set up with multiple namespaces","namespaces":"nifi"}
...
{"level":"info","ts":1648403625.282073,"logger":"controller.nifidataflow","msg":"Starting workers","reconciler group":"nifi.konpyutaika.com","reconciler kind":"NifiDataflow","worker count":1}
{"level":"info","ts":1648403877.1215262,"logger":"controllers.NifiCluster","msg":"CR status updated","status":"ClusterInitializing"}
{"level":"info","ts":1648403877.1492836,"logger":"controllers.NifiCluster","msg":"Nifi cluster state updated"}
{"level":"info","ts":1648403877.1786754,"logger":"controllers.NifiCluster","msg":"CR status updated","status":"ClusterInitialized"}
{"level":"info","ts":1648403877.2189906,"logger":"controllers.NifiCluster","msg":"CR status updated","status":"ClusterReconciling"}
{"level":"info","ts":1648403877.2191544,"logger":"controllers.NifiCluster","msg":"Reconciling cert-manager PKI","component":"nifi","clusterName":"tlsnifi","clusterNamespace":"nifi"}
{"level":"info","ts":1648403877.3570118,"logger":"controllers.NifiCluster","msg":"resource created","component":"nifi","clusterName":"tlsnifi","clusterNamespace":"nifi","kind":"*v1.Service","name":"tlsnifi-headless"}
{"level":"info","ts":1648403877.3572803,"logger":"controllers.NifiCluster","msg":"A new resource was not found or may not be ready"}
{"level":"info","ts":1648403877.3573396,"logger":"controllers.NifiCluster","msg":"server secret not ready: Secret \"tlsnifi-0-server-certificate\" not found"}
{"level":"info","ts":1648403877.3895488,"logger":"controllers.NifiCluster","msg":"CR status updated","status":"ClusterReconciling"}
{"level":"info","ts":1648403877.3896518,"logger":"controllers.NifiCluster","msg":"Reconciling cert-manager PKI","component":"nifi","clusterName":"tlsnifi","clusterNamespace":"nifi"}
{"level":"info","ts":1648403877.391266,"logger":"controllers.NifiCluster","msg":"resource updated","component":"nifi","clusterName":"tlsnifi","clusterNamespace":"nifi","kind":"*v1.Service","name":"tlsnifi-headless"}
{"level":"info","ts":1648403877.3913736,"logger":"controllers.NifiCluster","msg":"A new resource was not found or may not be ready"}
{"level":"info","ts":1648403877.3913915,"logger":"controllers.NifiCluster","msg":"server secret not ready: Secret \"tlsnifi-0-server-certificate\" not found"}
{"level":"info","ts":1648403877.6715496,"logger":"controllers.NifiUser","msg":"generated secret not found, may not be ready"}
{"level":"info","ts":1648403877.796963,"logger":"controllers.NifiUser","msg":"failed to reconcile user secret"}
{"level":"error","ts":1648403877.7970812,"logger":"controller.nifiuser","msg":"Reconciler error","reconciler group":"nifi.konpyutaika.com","reconciler kind":"NifiUser","name":"tlsnifi-0-node.tlsnifi-headless.nifi.svc.c2.bugslifesolutions.com","namespace":"nifi","error":"could not create user certificate: admission webhook \"webhook.cert-manager.io\" denied the request: spec.commonName: Too long: must have at most 64 bytes","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":1648403877.7981198,"logger":"controllers.NifiUser","msg":"generated secret not found, may not be ready"}
{"level":"info","ts":1648403877.8216078,"logger":"controllers.NifiUser","msg":"failed to reconcile user secret"}
{"level":"error","ts":1648403877.8217096,"logger":"controller.nifiuser","msg":"Reconciler error","reconciler group":"nifi.konpyutaika.com","reconciler kind":"NifiUser","name":"tlsnifi-0-node.tlsnifi-headless.nifi.svc.c2.bugslifesolutions.com","namespace":"nifi","error":"could not create secret with jks password: secrets \"tlsnifi-0-server-certificate\" already exists","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
The Cert-Manager pod logs indicate an issue with the Domain of the Nifikop tlsnifi-controller owned CertificateRequest:
E0327 18:10:13.819600 1 sync.go:270] cert-manager/orders "msg"="failed to create Order resource due to bad request, marking Order as failed" "error"="400 urn:ietf:params:acme:error:rejectedIdentifier: Error creating new order :: Cannot issue for \"tlsnifi-controller\": Domain name needs at least one dot" "resource_kind"="Order" "resource_name"="tlsnifi-controller-hjpcs-2838947340" "resource_namespace"="nifi" "resource_version"="v1"
in context:
I0327 18:09:00.095912 1 setup.go:202] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging-account-private-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1"
I0327 18:09:05.034646 1 setup.go:202] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging-account-private-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1"
I0327 18:10:03.378491 1 conditions.go:201] Setting lastTransitionTime for Certificate "tlsnifi-controller" condition "Ready" to 2022-03-27 18:10:03.37839815 +0000 UTC m=+63.713073736
I0327 18:10:03.378861 1 trigger_controller.go:181] cert-manager/certificates-trigger "msg"="Certificate must be re-issued" "key"="nifi/tlsnifi-controller" "message"="Issuing certificate as Secret does not contain a private key" "reason"="MissingData"
I0327 18:10:03.378913 1 conditions.go:201] Setting lastTransitionTime for Certificate "tlsnifi-controller" condition "Issuing" to 2022-03-27 18:10:03.378908921 +0000 UTC m=+63.713584482
I0327 18:10:03.762733 1 controller.go:161] cert-manager/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "key"="nifi/tlsnifi-controller" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"tlsnifi-controller\": the object has been modified; please apply your changes to the latest version and try again"
I0327 18:10:03.762897 1 conditions.go:201] Setting lastTransitionTime for Certificate "tlsnifi-controller" condition "Ready" to 2022-03-27 18:10:03.762888653 +0000 UTC m=+64.097564227
I0327 18:10:12.499043 1 controller.go:161] cert-manager/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "key"="nifi/tlsnifi-controller" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"tlsnifi-controller\": the object has been modified; please apply your changes to the latest version and try again"
I0327 18:10:12.591206 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "tlsnifi-controller-hjpcs" condition "Approved" to 2022-03-27 18:10:12.591190353 +0000 UTC m=+72.925865914
I0327 18:10:12.670590 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "tlsnifi-controller-hjpcs" condition "Ready" to 2022-03-27 18:10:12.670574343 +0000 UTC m=+73.005249913
E0327 18:10:13.819600 1 sync.go:270] cert-manager/orders "msg"="failed to create Order resource due to bad request, marking Order as failed" "error"="400 urn:ietf:params:acme:error:rejectedIdentifier: Error creating new order :: Cannot issue for \"tlsnifi-controller\": Domain name needs at least one dot" "resource_kind"="Order" "resource_name"="tlsnifi-controller-hjpcs-2838947340" "resource_namespace"="nifi" "resource_version"="v1"
I0327 18:10:13.938765 1 conditions.go:190] Found status change for Certificate "tlsnifi-controller" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2022-03-27 18:10:13.938746383 +0000 UTC m=+74.273421949
I0327 18:10:13.965550 1 trigger_controller.go:160] cert-manager/certificates-trigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="nifi/tlsnifi-controller" "retry_delay"=3599034548569
I0327 18:10:14.023466 1 trigger_controller.go:160] cert-manager/certificates-trigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="nifi/tlsnifi-controller" "retry_delay"=3598976585445
Here is the CertificateRequest:
apiVersion: cert-manager.io/v1
kind: CertificateRequest
metadata:
annotations:
cert-manager.io/certificate-name: tlsnifi-controller
cert-manager.io/certificate-revision: "1"
cert-manager.io/private-key-secret-name: tlsnifi-controller-s5j8v
creationTimestamp: "2022-03-27T18:10:12Z"
generateName: tlsnifi-controller-
generation: 1
name: tlsnifi-controller-hjpcs
namespace: nifi
ownerReferences:
- apiVersion: cert-manager.io/v1
blockOwnerDeletion: true
controller: true
kind: Certificate
name: tlsnifi-controller
uid: 83fecd4e-6f95-45f8-9d9c-612186ce9eb1
resourceVersion: "23860749"
uid: c7100e70-6c6e-404a-a9b6-3839d5227795
spec:
extra:
authentication.kubernetes.io/pod-name:
- cert-manager-786b9c87f8-nsb8z
authentication.kubernetes.io/pod-uid:
- 035536c1-97a2-49c3-84a8-f64e76c23e98
groups:
- system:serviceaccounts
- system:serviceaccounts:cert-manager
- system:authenticated
issuerRef:
group: cert-manager.io
kind: ClusterIssuer
name: letsencrypt-staging
request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJRStEQ0NBdUFDQVFBd0hURWJNQmtHQTFVRUF4TVNkR3h6Ym1sbWFTMWpiMjUwY205c2JHVnlNSUlDSWpBTgpCZ2txaGtpRzl3MEJBUUVGQUFPQ0FnOEFNSUlDQ2dLQ0FnRUF2bWlXLzZvVTNSMmVUK080M25HdmkwT3BoYUdICldHSUNreU5hT0UxNVEwV3dsSmF4dXFpMVZXcG9RTGVBL2liQUpqaEpJWlFMSGErUkpnN3piWWdYUFB2cnkzcG0KRmhjcC94NG95WVBNSFhIdytseFdzYlMvb2l0a2g5OWJMWFA1aWNFRTlGMW9nQWpTNTFCWXpUdDZNWDFYQmdSYwpkZ0RRMUVTdFFaS3p3WTF6UERqSHUwTW90eUh4RXg4bVlmNlUwY1pQa0owYkpDUkdob1dZVUZ5alFMMmROK1QxClY1dDdBdG9zSUhRYW1sTHIxWXZmZy9OYVljRHNOWjNIaWVUTTZNM1NsWWZ3TlE5eTFzMTRWVks2Nzl4bEJJUFcKd0xHYWo5SmVmZWZsa0V0MXQ3a2xqK2kvRUZGb05xcERQOXVFajhFTVJsNjA5S2xRYzkzUC9vaTdqUi9TdXFZdwp1Q1NWeVJBVElSMjd5eSsvU0VSUnRuYnc0Y3NHbkJ0SFA4czVlMTdxVGdkak02QjZLblFTWSs4NEhtUFh1cGRGCjBuZDhWUXFsZTl6a2hYVzh1VTN2K1k3RXZUdytUTVVLTDFjZTBsdDNxaG45ZkpsYUsweE8yT2w3UFk2N2xjcGMKdG9KdlBDSXhBV2doMXEwQkE4ZzJyV0gzRGV3VzhuZDYvMmNyN241ZXA3ZUl5aUlLSHFBb3VkUDBENDJ3UlRSUwpsaFYyUlV1VnZ4aUplUmdtbWp2VCtXWi9PWnRUa2ZNUkJCVjNtYUs4VUNOYzBUejhReWhzdWpnejIzcU5vakt4CjBZNHgvWFlhYVE1MitZY1ZUUE9IWHdCdmwwWUVzb0h4M08ydDBuaWI4Tk5BelRVbzIwWHgweXhDOC9yZ3gzSEcKVDdsaHRpcExpUUFBc3JVQ0F3RUFBYUNCbFRDQmtnWUpLb1pJaHZjTkFRa09NWUdFTUlHQk1GTUdBMVVkRVFSTQpNRXFDRW5Sc2MyNXBabWt0WTI5dWRISnZiR3hsY29ZMGMzQnBabVpsT2k4dmRHeHpibWxtYVM5dWN5OXVhV1pwCkwyNXBabWwxYzJWeUwzUnNjMjVwWm1rdFkyOXVkSEp2Ykd4bGNqQUxCZ05WSFE4RUJBTUNBQUF3SFFZRFZSMGwKQkJZd0ZBWUlLd1lCQlFVSEF3SUdDQ3NHQVFVRkJ3TUJNQTBHQ1NxR1NJYjNEUUVCRFFVQUE0SUNBUUNxUGxBTApNaGRSZzBRaVFtNzQwKy9GQXJicjRqUjVoSWZXelhXN1pON0xqbXBmMkJSZlpnbUNHOFdRQVMzWWtQeXFicVdOCitubTcwYWRZdVAweHNFTHEwRi9aL096R05kU0J0OEdVSHJ2di9NalYzVUl4bjVKUDgyYTBVcXpibXU5VVhOMHgKMXFnOElQcDVuMTBoSUJmWDViK1Jabm9pYnowcmtjcVNqMjUyTnZZd3lkNmt6cTV2ejRnQ2hjVW9CZEhOeXBkMgphL0dxQWkrYm1RUSswanJyejJ6dUppNC9JUlF5eTNCT1FSRW5tVDJCMlh5YVhtQnJtZ1hmZytTbGdzVWN0NEFhClE2MXBFQUVKT3F3aWtVbmZXbndqcnh5Y0xoYkR0aGo0R0drZUgzdmplbWtkbENBMWdxSTJGVmU5WWJpWkt6KzIKNlg4bGxRNlAzaDhyY1pNbVRUcUl1eldIMFEzZVBUWTdtZGt1YW1hR1h2TisvM3hkTlNiaFZuckwycHNyTFFXUwpxM096aXB1WFM5UFAySDk2cVplSy8wWlpDWFhpNXc4czlPZW1RdUtFcXF3RkJIYXdSckY1SE5UTWk1ekVuTjZpCm1xZ2trNmdZZmk0L2RFRFdSeEJ6enFxMHdPOThRUkxsUUkxOVQ5R1JzMk1WaXpBM0pNUUNlcHRnVlR3R3lDUnkKN3pqNkQwUEZ5Z0gvMmt5RGlsZ0NGcUVSYkd1K2R3RmhmejhFNUIwVElmVC9DSXlhYWtmMFA4N3VQVnp4N0hEVwpCaDZEckxmbWRlRDNjR1htL2tPT09NT21OaWkyZ0FFVzh3RTBkQWdRam9BbkFSUE1VMDNIcE5QeFJhQXVmZXFTCmJ4cm5CdmZtSHhrVnlwd01abjJDYTJQVEZvbmxOREozZk1hYmtBPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==
uid: 62f7108a-f2ba-4b63-b558-e52ec01a78aa
usages:
- client auth
- server auth
username: system:serviceaccount:cert-manager:cert-manager
status:
conditions:
- lastTransitionTime: "2022-03-27T18:10:12Z"
message: Certificate request has been approved by cert-manager.io
reason: cert-manager.io
status: "True"
type: Approved
- lastTransitionTime: "2022-03-27T18:10:12Z"
message: 'Failed to wait for order resource "tlsnifi-controller-hjpcs-2838947340"
to become ready: order is in "errored" state: Failed to create Order: 400 urn:ietf:params:acme:error:rejectedIdentifier:
Error creating new order :: Cannot issue for "tlsnifi-controller": Domain name
needs at least one dot'
reason: Failed
status: "False"
type: Ready
failureTime: "2022-03-27T18:10:13Z"
Environment
- nifikop version:
apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/psp: privileged
creationTimestamp: "2022-03-27T17:53:40Z"
generateName: nifikop-5d7d6fb5cd-
labels:
app: nifikop
control-plane: nifikop
name: nifikop
operator: nifi
pod-template-hash: 5d7d6fb5cd
release: nifikop
name: nifikop-5d7d6fb5cd-trjll
namespace: nifi
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: nifikop-5d7d6fb5cd
uid: 4e13caad-5cae-47f4-82fc-93b1b391beec
resourceVersion: "23854732"
uid: 8b09a927-397d-432b-a6d5-7a0264e6324e
spec:
containers:
- args:
- metrics-bind-address=8081
- --leader-elect
- --cert-manager-enabled=true
command:
- /manager
env:
- name: WATCH_NAMESPACE
value: nifi
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: OPERATOR_NAME
value: nifikop
- name: LOG_LEVEL
value: Info
image: ghcr.io/konpyutaika/docker-images/nifikop:v0.10.0-release
imagePullPolicy: Always
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8081
scheme: HTTP
initialDelaySeconds: 15
periodSeconds: 20
successThreshold: 1
timeoutSeconds: 1
name: nifikop
ports:
- containerPort: 8081
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /readyz
port: 8081
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 250m
memory: 256Mi
requests:
cpu: 250m
memory: 256Mi
securityContext:
allowPrivilegeEscalation: false
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-rh9rl
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: worker5
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
runAsUser: 1000
serviceAccount: nifikop
serviceAccountName: nifikop
terminationGracePeriodSeconds: 10
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: kube-api-access-rh9rl
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2022-03-27T17:53:40Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2022-03-27T17:53:50Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2022-03-27T17:53:50Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2022-03-27T17:53:40Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://e9836ba7f894653319d3d8fed0549270b6323083f04330b54b909cb775a7b782
image: ghcr.io/konpyutaika/docker-images/nifikop:v0.10.0-release
imageID: ghcr.io/konpyutaika/docker-images/nifikop@sha256:99a59041be7cf06e69f4112b9e1e2c6ee6a0320a7942680bc7daf1d13da0a6be
lastState: {}
name: nifikop
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2022-03-27T17:53:43Z"
hostIP: 10.5.2.61
phase: Running
podIP: 10.244.12.80
podIPs:
- ip: 10.244.12.80
qosClass: Guaranteed
startTime: "2022-03-27T17:53:40Z"
- go version:
- Kubernetes version information:
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:34:54Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
-
Kubernetes cluster kind:
-
NiFi version: apache/nifi:1.12.1
Possible Solution
Suffix the auto-created controller Common Name/Domain with the NifiCluster's spec.listenersConfig.clusterDomain
eg. tlsnifi-controller.c2.bugslifesolutions.com
Pattern: <metadata.name>-controller.<listenersConfig.clusterDomain>
Additional context
https://github.com/konpyutaika/nifikop/blob/b9f6cec46568e1ea6a3243f9b0c0565ffea93050/api/v1alpha1/nificluster_types.go#L711
Seems to have a work-around via setting Spec.NifiControllerTemplate to a value.
I'll give it a try...
The override worked as expected..
I0327 23:01:31.775293 1 trigger_controller.go:181] cert-manager/certificates-trigger "msg"="Certificate must be re-issued" "key"="nifi/tlsnifi.c2.bugslifesolutions.com" "message"="Issuing certificate as Secret does not contain a private key" "reason"="MissingData"
I0327 23:01:31.775286 1 conditions.go:201] Setting lastTransitionTime for Certificate "tlsnifi.c2.bugslifesolutions.com" condition "Ready" to 2022-03-27 23:01:31.775224688 +0000 UTC m=+17552.109900271
I0327 23:01:31.775427 1 conditions.go:201] Setting lastTransitionTime for Certificate "tlsnifi.c2.bugslifesolutions.com" condition "Issuing" to 2022-03-27 23:01:31.775418876 +0000 UTC m=+17552.110094455
I0327 23:01:32.196356 1 controller.go:161] cert-manager/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "key"="nifi/tlsnifi.c2.bugslifesolutions.com" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"tlsnifi.c2.bugslifesolutions.com\": the object has been modified; please apply your changes to the latest version and try again"
I0327 23:01:32.196503 1 conditions.go:201] Setting lastTransitionTime for Certificate "tlsnifi.c2.bugslifesolutions.com" condition "Ready" to 2022-03-27 23:01:32.19649708 +0000 UTC m=+17552.531172641
I0327 23:01:37.111923 1 controller.go:161] cert-manager/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "key"="nifi/tlsnifi.c2.bugslifesolutions.com" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"tlsnifi.c2.bugslifesolutions.com\": the object has been modified; please apply your changes to the latest version and try again"
I0327 23:01:37.201653 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "tlsnifi.c2.bugslifesolutions.com-kq4qb" condition "Approved" to 2022-03-27 23:01:37.201638867 +0000 UTC m=+17557.536314430
I0327 23:01:37.273872 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "tlsnifi.c2.bugslifesolutions.com-kq4qb" condition "Ready" to 2022-03-27 23:01:37.273860197 +0000 UTC m=+17557.608535757
@bugslifesolutions how did you fix the problem?
You can influence the identities that nifikop assigns to the controller (the identity for the operator itself) and nifi nodes via various NifiCluster properties, all of which are documented here: https://konpyutaika.github.io/nifikop/docs/v1.9.0/5_references/1_nifi_cluster/
Sometimes the default template can result in invalid names (usually too long in my experience), so overriding these templates is how you fix that.
Specifically, see:
-
nifiControllerTemplate -
nodeUserIdentityTemplate
