k8s-infra Fix OCP HSTS errors

Issue

When we access the ocp console, then we got such a message from google chrome

You cannot visit oauth-openshift.apps.ocp.snowdrop.dev right now because the website uses HSTS. Network errors and attacks are usually temporary, so this page will probably work later.

HSTS: https://developer.mozilla.org/en-US/docs/Glossary/HSTS

Temporary workaround is to type thisisunsafe within the google chrome window - https://stackoverflow.com/questions/33268264/chromethe-website-uses-hsts-network-errors-this-page-will-probably-work-late

Mar 29 '23 15:03 jacobdotcosta

Dont forget to add the link to the issue/PR to track that from your tasks ;-)

Apr 11 '23 08:04 cmoulliard

Can you add the link to the ticket created please ? @jacobdotcosta

May 25 '23 10:05 cmoulliard

https://redhat.service-now.com/surl.do?n=RITM1483142

@cmoulliard

May 29 '23 09:05 jacobdotcosta

I think that I know how to fix it.

We dont have issues when we access https://console-openshift-console.snowdrop-eu-de-1-bx2-4x16-0c576f1a70d464f092d8591997631748-0000.eu-de.containers.appdomain.cloud/ as their HTTPS certificate has been signed for the domain *.cloud by Let's encrypt - https://letsencrypt.org/certificates/#intermediate-certificates

instead the certificate generated by the ocp ingress operator for us

So, if we ask to Let's encrypt and Godaddy to sign a Certificate request for *.apps.ocp.snowdrop.dev, that should work.

The next challenge will be to pass such a certificate within the Openstack ocp install config file. Maybe using: https://docs.openshift.com/container-platform/4.13/installing/installing_openstack/installing-openstack-installer-custom.html#installation-osp-describing-cloud-parameters_installing-openstack-installer-custom

WDYT ? @jacobdotcosta

May 30 '23 14:05 cmoulliard

I tested manually this procedure after updating the certificate and issuer role of k8s-infra, collected the updated secret and patched as described here: https://docs.openshift.com/container-platform/4.13/security/certificates/replacing-default-ingress-certificate.html#replacing-default-ingress_replacing-default-ingress

Now, the access to https://console-openshift-console.apps.ocp.snowdrop.dev/dashboards do not generate a HTST error ;-)

oc login is reporting nevertheless an error

oc login --token=sha256~xxxx --server=https://api.ocp.snowdrop.dev:6443
error: x509: “kube-apiserver-lb-signer” certificate is not trusted

FYI: @jacobdotcosta

May 30 '23 16:05 cmoulliard

A part of the problem could be automated if we pass to the openshift installer config file the parameter additionalTrustBundle See how that works here: https://github.com/openshift/installer/blob/master/docs/user/customization.md#proxy which corresponds to what it is described here: https://docs.openshift.com/container-platform/4.13/security/certificates/replacing-default-ingress-certificate.html#replacing-default-ingress_replacing-default-ingress (create a custom CA secret and patch proxy/cluster)

NOTE: We need of course to populate first and separately the CA crt and keys before to create a cluster !

@jacobdotcosta

Nov 21 '23 10:11 cmoulliard

I still have an issue using such certificate/issuer

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-prod-qshift-snowdrop-dev
  labels:
    app: qshift-ca-cert
  namespace: snowdrop-site
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email:
    privateKeySecretRef:
      name: letsencrypt-prod-snowdrop-dev
    solvers:
      - dns01:
          webhook:
            config:
              apiKeySecretRef:
                name: godaddy-api-key
                key: token
              production: true
              ttl: 600
            groupName: acme.mycompany.com
            solverName: godaddy
        selector:
          dnsNames:
            - 'api.qshift.snowdrop.dev'
            - 'oauth-openshift.apps.qshift.snowdrop.dev'
            - 'console-openshift-console.apps.qshift.snowdrop.dev'
            - 'grafana-openshift-monitoring.apps.qshift.snowdrop.dev'
            - 'prometheus-k8s-openshift-monitoring.apps.qshift.snowdrop.dev'
            - 'integrated-oauth-server-openshift-authentication.apps.qshift.snowdrop.dev'
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: qshift-snowdrop-dev
  labels:
    app: qshift-ca-cert
  namespace: snowdrop-site
spec:
  renewBefore: 2136h
  duration: 2190h
  privateKey:
    size: 2048
    algorithm: RSA
  issuerRef:
    kind: Issuer
    name: letsencrypt-prod-qshift-snowdrop-dev
  secretName: qshift-snowdrop-dev-tls
  dnsNames:
    - 'api.qshift.snowdrop.dev'
    - 'oauth-openshift.apps.qshift.snowdrop.dev'
    - 'console-openshift-console.apps.qshift.snowdrop.dev'
    - 'grafana-openshift-monitoring.apps.qshift.snowdrop.dev'
    - 'prometheus-k8s-openshift-monitoring.apps.qshift.snowdrop.dev'
    - 'integrated-oauth-server-openshift-authentication.apps.qshift.snowdrop.dev'

Nov 21 '23 18:11 cmoulliard

Problem fixed after issuing the following certificate

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-prod-qshift-snowdrop-dev
  labels:
    app: qshift-ca-cert
  namespace: snowdrop-site
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email:
    privateKeySecretRef:
      name: letsencrypt-prod-snowdrop-dev
    solvers:
      - dns01:
          webhook:
            config:
              apiKeySecretRef:
                name: godaddy-api-key
                key: token
              production: true
              ttl: 600
            groupName: acme.mycompany.com
            solverName: godaddy
        selector:
          dnsNames:
            - '*.apps.qshift.snowdrop.dev'
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: qshift-snowdrop-dev
  labels:
    app: qshift-ca-cert
  namespace: snowdrop-site
spec:
  renewBefore: 2136h
  duration: 2190h
  privateKey:
    size: 2048
    algorithm: RSA
  issuerRef:
    kind: Issuer
    name: letsencrypt-prod-qshift-snowdrop-dev
  secretName: qshift-snowdrop-dev-tls
  dnsNames:
    - '*.apps.qshift.snowdrop.dev'

Nov 22 '23 09:11 cmoulliard

Question from Antonio: To install the new cert manager on the OCP cluster we would still need the one on the snowdrop k8s server right?

Charles's response: No.

It is not needed to centralize everything on a cluster EXCEPT when rotation will take place and when secrets (= ca, tls) are still used.

So if we don't centralize, that means that we need to find a way to "copy/scp" the secret or rotated secret to the target cluster
If we install a cert manager / cluster, then it can also manage the rotation and save locally the secret. If the cluster is then removed, no need anymore to rotate the cert as needed by letsencrypt after 3months

@jacobdotcosta

Nov 22 '23 09:11 cmoulliard

k8s-infra k8s-infra copied to clipboard

Fix OCP HSTS errors

Issue

k8s-infra
k8s-infra copied to clipboard