Custom CA certificates in Postgres Operator UI

[keliansb]

• Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.8.2
• Where do you run it - cloud or metal? Kubernetes or OpenShift? Bare Metal K8s (RKE)
• Are you running Postgres Operator in production? We are currently deploying it
• Type of issue? Feature request

We are using Ceph Object Gateway as S3 compatible storage for WAL archiving. When trying to access the Backups menu in the UI, the following error appears in log file :

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)

The following extraEnvs was set in the Helm chart :

extraEnvs:
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        key: AWS_ACCESS_KEY_ID
        name: postgres-pod-secrets
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        key: AWS_SECRET_ACCESS_KEY
        name: postgres-pod-secrets
  - name: AWS_ENDPOINT
    valueFrom:
      configMapKeyRef:
        key: AWS_ENDPOINT
        name: postgres-pod-config
  - name: AWS_S3_FORCE_PATH_STYLE
    valueFrom:
      configMapKeyRef:
        key: AWS_S3_FORCE_PATH_STYLE
        name: postgres-pod-config
  - name: SPILO_S3_BACKUP_BUCKET
    value: postgres-operator-wal

The backup from a Postgres cluster works fine thanks to the environment variable WALG_S3_CA_CERT_FILE, but I don't find any way to set a custom ca certificate in the Postgres Operator UI.

[digitalnerd]

Didn't use Postgres Operator UI before but you can set your custom CA certificate into postgres CR as mention about that here - https://github.com/zalando/postgres-operator/blob/master/docs/user.md#custom-tls-certificates

apiVersion: "acid.zalan.do/v1"
kind: postgresql

metadata:
  name: acid-test-cluster
spec:
  tls:
    secretName: "pg-tls"
    caFile: "ca.crt" # add this if the secret is configured with a CA

[keliansb]

Yes my postgres CR is already configured with a custom CA certificate, but it doesn't work.

[digitalnerd]

Looks like you have misconfiguration somewhere. Could you present what you've done regarding certs and CR configuration, please?

[keliansb]

From what I understand the backup tab of the UI does not depend on the postgres CR, but rather it is a global configuration related to S3 storage, right? It seems that the tls.caFile parameter is only for PostgreSQL connections.

[digitalnerd]

A CA certificate is about interconnections between server and clients (SSL/TLS, mTLS). For S3 storage you have to configure another parameters based on a ConfigMap. Like this, it's just an example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-pod-config
  namespace: postgres-cluster
data:
  BACKUP_SCHEDULE: "*/5 * * * *"
  BACKUP_NUM_TO_RETAIN: "10"
  USE_WALG_BACKUP: "true"
  AWS_ENDPOINT: "https://...."
  WAL_S3_BUCKET: "name-of-a-bucket"
  AWS_ACCESS_KEY_ID: <ACCESS_KEY_ID>
  AWS_SECRET_ACCESS_KEY: <SECRET_KEY_ID>

If I understood you correctly.

[keliansb]

Yes my ConfigMap/Secret for S3 storage is working for Postgres Operator, but for the UI we can't reference them. Instead we need to define extraEnvs, like the ones I mentioned in my original post. For the Postgres Operator I'm using the environment variable WALG_S3_CA_CERT_FILE to reference a custom CA certificate, but it doesn't work for the UI.

[digitalnerd]

Aha, I didn't read your post properly. I thought you need to configure e2e encryption between server and client based on the certificates (verify-full mode in Postgres). But I didn't work with your case unfortunately, sorry.

[stgarf-sx]

Chiming in here -- I am also unable to set the CA Certificate using either base64 encrypted string or using path to file for the postgres-operator-ui.

[cstisa]

Hi, Do you have news on this ? I managed to have this working using REQUESTS_CA_BUNDLE (request lib) and SSL_CERT_FILE (flash gvent lib) environment variables as extraEnvs

Best Regards