New internal certificate authority not loaded by running pods

[sguyennet]

The certificate authority for internal certificate is mounted with a subpath. Here is an example with the registry deployment but all components are impacted: https://github.com/goharbor/harbor-operator/blob/master/controllers/goharbor/registry/deployments.go#L189

This is an issue because when cert-manager automatically renew the internal certificate, the ca.crt is not updated inside the pods and the communication between the different Harbor components fails till you restart the pods manually.

This is a known limitation of the subpath feature: https://github.com/kubernetes/kubernetes/issues/50345

[steven-zou]

@bitsf

Is this bug related to the issues you fixed before? Please take a look.

[bitsf]

maybe we need find a way to watch the configmap change

[bitsf]

@sguyennet do you have any idea about this issue ?

[BRONSOLO]

@bitsf @steven-zou could someone provide a link to the PR that resolved this issue? I'm curious to see what the fix for this was in the end.

[bitsf]

PR https://github.com/goharbor/harbor-operator/pull/876 is supposed to fixed this

[thcdrt]

@bitsf I think there is a misunderstanding here, I don't see the relation between this issue and the PR #876. What does it fix regarding automatic reload of certificates inside Harbor pods ?

[charandas]

I was thinking of using something like Reloader but am blocked in that there is no way to set metadata.annotations on core components, only the pod template's metadata annotations can be currently set as provided by this PR.

I need to be able to set the annotations on deployment's metadata itself. Something like:

secret.reloader.stakater.com/reload: some-cert-related-secret

[charandas]

After looking at the original issue in the description which describes a limitation of K8s with subPath use in secrets-based volume mounts, I thought I will check whether harbor-operator based deployments use subPath. They do, and it is for ca.crt.

    volumeMounts:
    - mountPath: /etc/chartmuseum
      name: config
    - mountPath: /harbor_cust_cert/ca.crt
      name: internal-certificates
      readOnly: true
      subPath: ca.crt

Subpath based volume mounts can be refreshed in K8s, it's a know limitation and is documented. A pod restart would work, or another recommended solution is to mount the whole secret (as a directory) elsewhere and symlink to the target file in that directory from where its needed.

We can do the symlinking in the initContainer or have the symlink be built into the /harbor_cust_cert/ca.crt file in the base container image. Until then, a pod restart will indeed be needed.

[charandas]

I am not even sure why the same secret is being mounted both to /etc/{{component_name}}/ssl and /harbor_cust_cert. It's for the latter that the subpath usage exists, so just removing that second mount and referencing the CA from /etc/{{component_name}}/ssl might be enough, but I don't know how /harbor_cust_cert directory is being used.

Grepping for /harbor_cust_cert and InternalCertificateAuthorityDirectory doesn't bring up much except the place where the volumeMount configs are being defined.

[charandas]

Ha! It's documented here: https://goharbor.io/docs/2.4.0/administration/vulnerability-scanning/configure-custom-certs/

/harbor_cust_cert has a special place in Harbor Configuration. This will mean that our our SSL cert issues could be resolved if we simply symlinked /harbor_cust_cert to /etc/{{component_name}}/ssl.

[sagikazarmark]

@charandas if I understand correctly either using reloader (either with auto config or somehow injecting the annotations) or changing the mount options is the solution.

@steven-zou @bitsf can you please reopen this issue?

[thcdrt]

The simplest would probably to change the mount option to remove the subpath but we need to:

• Check certificates are hot reloaded by Harbor components, without needing to restart application
• Split some secrets to not mount useless certs or keys everywhere

[thcdrt]

Well, we did some checks with @sguyennet and found that Harbor components trust ca on boot by launching this script: https://github.com/goharbor/harbor/blob/main/make/photon/common/install_cert.sh

So if we want to remove subpath and let Harbor hot reload certificates, we need CA certificates to have a long valid duration.

[sagikazarmark]

Funny thing is, I never experienced this with anything, but ChartMuseum. Although it's deprecated, when ChartMuseum communication fails due to a cert issue, project list requests also completely fail (at least on the portal). So I'd say this is a rather serious issue, even if it only affects ChartMuseum (which I'm not sure about).

As a temporary solution, I can absolutely live with using Reloader to restart workloads, but as @charandas mentioned, that requires annotations to be propagated from top level CRDs to the actual workloads.