Unable to set Job .template.metadata.annotations

[npapapietro]

Overview

The metadata exposed on the CRD only applies the metadata labels to the .metadata on the Job but not its .spec.template.metadata of the Pod kicked off by the job when a migration occurs.

For example if I configure my cluster as:

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: cluster1
  namespace: someCluster
spec:
  metadata:
    annotations:
      sidecar.istio.io/inject: "false"
  // ...
  instances:
    - name: instance1
      metadata:
        annotations:
          sidecar.istio.io/inject: "false"
        // ...
  backups:
    pgbackrest:
      metadata:
        annotations:
          sidecar.istio.io/inject: "false"
        // ...

None of those make it to the job pod. As you see in my example, this has a strong use case for working with Istio

Code Links

Job.metadata https://github.com/CrunchyData/postgres-operator/blob/master/internal/controller/postgrescluster/volumes.go#L435

Job.metadata.spec.template.metadata https://github.com/CrunchyData/postgres-operator/blob/master/internal/controller/postgrescluster/volumes.go#L477

Environment

• Platform: EKS
• Platform Version: 1.23
• PGO Image Tag: ubi8-5.1.2-0
• Postgres Version: 12

[benjaminjb]

Seems like a reasonable idea, so I've got it in our backlog -- but would also welcome a PR for it!

[ohemelaar]

This would require adding something in the CRDs like PostgresCluster.spec.backups.pgbackrest.jobs.metadata right ?

[npapapietro]

I believe so. The workaround I used here to write a kyverno ClusterPolicy that attaches another container to all jobs that kills istio pod when all job pods successfully quit. My solution is specific to a cluster running kyverno. For reference I wrote https://github.com/skyfjell/istio-proxy-quit

[andrewlecuyer]

@npapapietro have you tried the Native Sidecar support that is now included in Istio?

• https://istio.io/latest/blog/2023/native-sidecars/
• https://istio.io/latest/docs/reference/commands/pilot-agent/
• https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/

I've been doing some testing, and this appears to allow Istio sidecars to work just fine with pgBackRest Jobs. Therefore, it should no longer be necessary to add sidecar.istio.io/inject: "false" to these Jobs.

[npapapietro]

I have not. I have moved to edge w/ k3s and no longer using istio. I appreciate the tip, will close out this issue with this new info.