pod in crashloop back

[julienroussel00]

I tried installing imageswap on one of the k8s cluster im in charge with and im getting the following error message (see logs below)... I can see the certificat request status is approved.

NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION imageswap.imageswap-system.cert-request 4m12s kubernetes.io/kubelet-serving system:serviceaccount:imageswap-system:imageswap-sa Approved

[2022-11-11 19:38:49,691] INFO: ImageSwap Init [2022-11-11 19:38:49,691] INFO: Starting TLS init process [2022-11-11 19:38:49,715] INFO: Did not find secret "imageswap-tls" in the "imageswap-system" namespace [2022-11-11 19:38:49,715] INFO: Generating new cert/key pair for TLS [2022-11-11 19:38:49,764] INFO: Building K8s CSR [2022-11-11 19:38:49,767] INFO: Looking for existing CSR [2022-11-11 19:38:49,788] INFO: Deleting k8s csr [2022-11-11 19:38:49,808] INFO: Existing certificate request deleted [2022-11-11 19:38:49,809] INFO: Create k8s CSR [2022-11-11 19:38:49,831] INFO: Certificate signing request "imageswap.imageswap-system.cert-request" has been created [2022-11-11 19:38:49,846] INFO: Patch k8s CSR: imageswap.imageswap-system.cert-request [2022-11-11 19:38:49,864] INFO: Certificate signing request "imageswap.imageswap-system.cert-request" is approved [2022-11-11 19:38:49,879] INFO: Waiting for certificate approval [2022-11-11 19:38:49,887] INFO: Waiting for certificate approval ...... [2022-11-11 19:38:54,832] INFO: Waiting for certificate approval [2022-11-11 19:38:54,839] INFO: Waiting for certificate approval [2022-11-11 19:38:54,849] INFO: Waiting for certificate approval [2022-11-11 19:38:54,858] INFO: Waiting for certificate approval [2022-11-11 19:38:54,868] INFO: Waiting for certificate approval [2022-11-11 19:38:54,868] INFO: Timed out reading certificate request "imageswap.imageswap-system.cert-request" Traceback (most recent call last): File "/app/imageswap-init.py", line 1201, in main() File "/app/imageswap-init.py", line 1190, in main init_tls_pair(imageswap_namespace_name) File "/app/imageswap-init.py", line 677, in init_tls_pair tls_pair = build_tls_pair( File "/app/imageswap-init.py", line 347, in build_tls_pair tls_cert_pem = get_tls_cert_from_request( File "/app/imageswap-init.py", line 319, in get_tls_cert_from_request tls_cert = base64.b64decode(k8s_csr.status.certificate) File "/usr/local/lib/python3.10/base64.py", line 80, in b64decode s = _bytes_from_decode_data(s) File "/usr/local/lib/python3.10/base64.py", line 45, in _bytes_from_decode_data raise TypeError("argument should be a bytes-like object or ASCII " TypeError: argument should be a bytes-like object or ASCII string, not 'NoneType'

[phenixblue]

Can you set logging to debug and post the logs from that? Also, did you check the contents of the certificate?

[dex4er]

I have the same error I believe EKS cannot issue the certificate if CertificateSigningRequest does not start with csr- prefix. Unfortunately its name is imageswap.imageswap-system.cert-request so it stuck on Approved.

[phenixblue]

This issue seems to indicate the opposite...that containing the csr- prefix will prevent the certificate from being issued.

https://github.com/aws/containers-roadmap/issues/2079

[dex4er]

Ah so as https://docs.aws.amazon.com/eks/latest/userguide/cert-signing.html explains, application cannot use signerName: kubernetes.io/kubelet-serving. There is however additional signerName: beta.eks.amazonaws.com/app-serving

I can confirm that after installing this kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - https://raw.githubusercontent.com/phenixblue/imageswap-webhook/master/deploy/install.yaml
patches:
  - patch: |-
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: imageswap-env
        namespace: imageswap-system
      data:
        IMAGESWAP_CSR_SIGNER_NAME: beta.eks.amazonaws.com/app-serving

the app starts and certificate is approved and issued.