3scale-operator icon indicating copy to clipboard operation
3scale-operator copied to clipboard
verified publisher icon 3scale

THREESCALE-11020 Redis TLS certs and keys for porta and backend

Open valerymo opened this issue 1 year ago • 11 comments

Jira: https://issues.redhat.com/browse/THREESCALE-11020

Add a way for the user to provide Redis TLS certs and keys for porta and backend

  • This PR enables Porta and Apisonator to load TLS configuration details for connecting to Redis. It introduces new environment variables that specify the locations of certificate files and indicate whether TLS mode is enabled.
  • The PR includes basic validation for Redis certificate-related fields and Redis URLs within the system-redis and backend-redis secrets,
  • Documentation has been updated to reflect these changes.

Validation

Prepare for validation

cd 3scale-operator
make install
export NAMESPACE=3scale-test
oc new-project $NAMESPACE
oc project $NAMESPACE
make download
make cluster/create/system-postgres

1. Install Redis Server for Test

- Copy the entire scripts block provided below, open your terminal and paste the script into the command line. This will create the Redis server, including following resources: - Secret: redis-tls-secret - ConfigMap: redis-config-redis - Deployment/pod: redis - Service: redis 

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata:
  name: redis-tls-secret
  namespace: 3scale-test
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lVYS9FaGY1UDR3WWVnYVZhaGpqT0R6ME1QVTJnd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF3d09NVGN5TGpNd0xqSXdOeTR4T1RFd0hoY05NalF4TVRJME1EZ3lNakF6V2hjTgpNalV4TVRJME1EZ3lNakF6V2pBWk1SY3dGUVlEVlFRRERBNHhOekl1TXpBdU1qQTNMakU1TVRDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUl0eld6OHcrTDBOYXAzN0FIcXR4Rjh6OEdXRFROUnoKOUxVa0JYcXU1SkZhc1lxejYrZ2k3S3dsWUlTeEFwVzd4NXprWDZ4bjBCeGxsMXVNdzExZHVyeDJNUjhlYXFxQQpxbElFVzZwTU1kTGNndkhJUit0TGhLRFVvaVpsajdKRkk1MGYvYjhUaXRKVTByY1dhYmd4SVA4QnlINUdxSkR2CmtFZGgvNC9SbTlYeWU4SmhuMEVqRzdPUWxTM1MvOGJHSGpoV2Zjdys2QnJidDI5TE1aTzdJRmFlRVh4azRaQjAKS1VNODJ0TFhxV1VwaUdxQUhuWncrZUtBSlZDOFBzUzB5NU9aTGt4TE9GUlJnajJRYkF3TlZ1emxBN0VMQ0U4dApyVkx3OWV3aEhsVkhUcVZ0UXYwNWZVb2ZqOURkb3Yxd3F2OG1uZXJ3OHBnRVR3Yi9RemY1VGRjQ0F3RUFBYU5UCk1GRXdIUVlEVlIwT0JCWUVGRGpocmlHaDZxbUZVS2Q0OG14UERUWi9CYThBTUI4R0ExVWRJd1FZTUJhQUZEamgKcmlHaDZxbUZVS2Q0OG14UERUWi9CYThBTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFDdCtPNTZnanZVTUo5MFdaekFFcEFLcTNFTDY3c1I4NXlYam5EeXZML1lRRjZZSzRhUmVaZ1JICms5WCs2cDNyd2tsQmF6MzFFTFNFSGlPUkVTWDJzNW9WSVlId01SNUpsOE5VZTB6NGVBZlNNcTNtQmFuYWxneWYKQktROHZRc1UxTGVJQkpGRVJ5dG5aVE03VThFbFVsUkZSZzAzVUcyVG5ibGNxalFRa1lTMmZYT3M1czBlMHI2eQpaektXVWZhemtzTlkzSHlUcUxRTDlEZUM5STd6R0s2c2E5dHhueTFOM0I5bWJHZ1l3dlFZNDlvTm1kem9oWW16CkxmVVZJeERSWUcwNnI1UldoTTVtZDhZNmJUSGo1TExHT1A2YlEvYWJteXZxbFpuVnZjWTZaandKUzhNWVJJSk8KMmxrczl3cXRjeENla3VJZFBrOUNsNlVBZ3E5TkJoVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  redis-server.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURBakNDQWVxZ0F3SUJBZ0lVYUIyZGNLRXJuRHM0MEVqMitwT210bmpRQytVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF3d09NVGN5TGpNd0xqSXdOeTR4T1RFd0hoY05NalF4TVRJME1EZ3lNakF6V2hjTgpNalV4TVRJME1EZ3lNakF6V2pBWk1SY3dGUVlEVlFRRERBNHhOekl1TXpBdU1qQTNMakU1TVRDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQVAxN2tGaHBRdDhpcUlRWEJOUEozbHBrOFFGUWJnUncKc2VlQkNoa1RTYTd5S0hvQS9VTHVHY3QzZHVubWZPeWdNeEhueW9LUmR0NDVnWWlDOGNnQjE4OWRDRHo2cy84ZAp2TUdoUFUrblhkOWxyVHNtYXpLNW9McnVETmI3TXhBVDFVdjNIMXkvbFErY2tPTkttLzVhWndoNnFxL25XVFZtCkdOZUVGcDJyNUczTkVXcTJXWm1KM0RCc3pMeUdFciszdUlJbFVsaUk2bXRZUGRRd1QwR3RqTDJpVldRVUd2Q2sKYWpHclNJQUVETVFCdFNYeE1yOTlzS2VhS21iVDJ1L09BOEJjYVlqbXdZVnN4VDIzczR2UFltaUhjTFVjVFE4MApTblBLSW03NDJrTzhLZE5Id1hjN1BpejhUTVB0WTU1QkU5ZlBIaGdueVNBMVZUeWNQR1cyZEMwQ0F3RUFBYU5DCk1FQXdIUVlEVlIwT0JCWUVGT3BwbncveVd2dFo2L0llVzhUUlhXeG1uMUtXTUI4R0ExVWRJd1FZTUJhQUZEamgKcmlHaDZxbUZVS2Q0OG14UERUWi9CYThBTUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFCWnhOZGpZR1Y4M1RBUwp6amMyNjZnc3gwRTB0cTRVUnRGWHdFeXZTYkxvY3BmK1V2OCtmTFBYZGF0WG1lRm5MdlJZbXg0VGR1UlltMFBYCk1BRUpqcEE3UXdlM1NsSmd5R2loNk1PVVdIRFdNZnhQZWJQWUh5ZFQ3TXBKVVJNWkxjZ0tvWlVENWNKdDc1bVIKMk51Z3BvdDdxQnYybzZGdmI5cmpNYSt1WFdJUEdSekhidjJ6eVFSR1J3SUpZcitKaDNWbXcyRSt0Vit0VEUwQQovWlBxOTIxYnlWYkREZHI2aDMyUnRRN1NJUnAwLzBnWFZablY3NTB2Rm9Ub2xOK2wwLzJOeEVCWmZNSzVFcEdkCkViSFFvN2dETHg0REljZlhqZVlxUkFuYjNLbzhXK0ZrQS8wQUdQS2NSN2RIL2daVnFHMngrRk5QazQ4YzBDQkwKR0R2RjlRYncKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  redis-server.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRRDllNUJZYVVMZklxaUUKRndUVHlkNWFaUEVCVUc0RWNMSG5nUW9aRTBtdThpaDZBUDFDN2huTGQzYnA1bnpzb0RNUjU4cUNrWGJlT1lHSQpndkhJQWRmUFhRZzgrclAvSGJ6Qm9UMVBwMTNmWmEwN0ptc3l1YUM2N2d6Vyt6TVFFOVZMOXg5Y3Y1VVBuSkRqClNwditXbWNJZXFxdjUxazFaaGpYaEJhZHErUnR6UkZxdGxtWmlkd3diTXk4aGhLL3Q3aUNKVkpZaU9wcldEM1UKTUU5QnJZeTlvbFZrRkJyd3BHb3hxMGlBQkF6RUFiVWw4VEsvZmJDbm1pcG0wOXJ2emdQQVhHbUk1c0dGYk1VOQp0N09MejJKb2gzQzFIRTBQTkVwenlpSnUrTnBEdkNuVFI4RjNPejRzL0V6RDdXT2VRUlBYeng0WUo4a2dOVlU4Cm5EeGx0blF0QWdNQkFBRUNnZ0VBRG5vaktWbUJyenJNZ3hiSmVNc2J2dS9xNzlkSElVdktiVjFxVlRwTHlBa2UKbExFL3hiWFJsVlJTWDFPQnFRWVJSS0dIYUdPa2RWYTFkalY4VjU3N1UyV04xZVcvcC85cnkyZEpHQ2FIN3YxZwpvbk0wUmlaaDdxc3Y0b3RnUkRmTnc5UHVYNTYxaGJtOGNLN1BML3k3eTdrdHpIUWJIVGlpakpTSHNpT2lIVDh1CitWc3oySzZmSnhHNUE2SE03bldvdlBuK1VydFd4OGd4OEJNQ1FOa3dUVnQvVmxzcE1wcUo1czdqeUNncGFQN2MKV0wrWnh2K2xUcFUwcnpJUTFZMWYvL0RSeUI3OVEycUl0dVhEdkJnNnI5ckExbytSV0JuNnA3WVA0T21SUE4vZQpiMVZZcG14c1BBL2ppQkhsRVFPZVJWbEVSYzhJSGVib2IxZFZ1Rm15b1FLQmdRRC9LeFlvVzNxaGNmUEYya2EvCmliZTkvKzlyQzNlTkdHUGwyaDVCeDJwK3k4dnlQb3NabmZ5STRKRzNtcXpVS1luSHFPVzhpTVpnR2ZOUlRNdVQKeVFSQitsVlFldHpOcDJUZCt4dWlCMUJmWHIyanowcVZvV3F4dFhrMklRWXFJN2NMSGZVZ2dGeUtsL2lRRGZlOQpwd0YwZ3c5Nm1vT012VWp1S1FmZjh2RTYxUUtCZ1FEK1R4SWZtc1VyY2VFdmpkeUdhNllweTBZcEx2OVhnSk0rCkxIc1ZmVTgzNXNWeVJaUDZrV1EwSGwyVCtHMDBqcStsU20vQ296cWNZeE5ObmtGMEhOdEw2SlJJWTBZZEExSFUKSmc1eTY3U2w0U0hHeWdYNTdrZEE4ajhyS1I4UXpmQ0RHT3Q5d2NwNjZ4MVlMSWEwRDYvV2h3elFOeXl3QzdnTwoxS3dvZlhyUCtRS0JnRzU4WUk2KzlYMWNVdnBUaGhpL2IvRDBGZDNhekR3cTJHNlpJRXJKSndLYUNjZnRidHQ3CnZmSWlrdFhXUW9sbkp3SnR6blB4SVR4UllEck9yc05oNGRjVHBzYy9POFpNZWU5b0lGSHJLdER3dTlwbkVsdHgKMWpuMll2S2VJQVkxQ3Jma2s5UXI0R1llWVlFMm14UGljVTNheGVRSGJYaU9LVHIrUnl1Z0RQVzFBb0dBZm5vdwoxMHNRR0tWUWkyZ1FiMElHcCs2Uy9GU0ZaYTFxalpkdHQ2aFV4OGFjR0ZNR1g2NERtZkFvTmpsdGhxQVlOeXFvCkhyTXpxU2VWS0JzM0RscHpybk1EbkdUVE1BYkFvYlF6cDNBV3JoRWp6VXdZWU03aTNTZ2R4b2R6RGRaK2NaVHAKT2VneG5hUmxPYjhiVjE0ZDQ2SFMrNU1WUkpEdmYyRENKbmtScFhFQ2dZRUFnVFB3MlVaSkJRVVdGc0NVRjczdgpGcWZSbzdZYmtweDAxcFlBcmV0ak41TTR6ZHZRUlZPQlFmVGRJTU9SbFJ4OVBnRitYL3UrT0szUzB1aS8xdWlECjhYU2QvMXFjS3ZOdUJ6ZnZNdTVGeVdKRHNCUGJaUHZuWnEzdTM5WktaYTNXNG92UE9VWlFERGlCTHNhM21CekwKcFdYNWFyVHNTazRSSDExQ28zRTN0akU9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
type: Opaque
EOF


cat << EOF | oc create -f -
apiVersion: v1
data:
  redis.conf: |+
    # redis.conf
    bind 0.0.0.0
    protected-mode no
    port 6379
    tls-port 6380
    tls-cert-file /etc/redis/certs/redis-server.crt
    tls-key-file /etc/redis/certs/redis-server.key
    tls-ca-cert-file /etc/redis/certs/ca.crt
    tls-auth-clients yes
    stop-writes-on-bgsave-error no
    save ""
kind: ConfigMap
metadata:
  name: redis-config-redis
EOF


cat << EOF | oc create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: quay.io/fedora/redis-6
        ports:
        - containerPort: 6379
        volumeMounts:
        - name: redis-config-volume
          mountPath: /etc/redis/redis.conf
          subPath: redis.conf
        - name: redis-tls-volume
          mountPath: /etc/redis/certs
          readOnly: true
        command: ["/bin/sh", "-c", "redis-server /etc/redis/redis.conf"]
      volumes:
      - name: redis-config-volume
        configMap:
          name: redis-config-redis
      - name: redis-tls-volume
        secret:
          secretName: redis-tls-secret
EOF


cat << EOF | oc create -f -
apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  ports:
    - port: 6379         # Non-TLS (unencrypted) port
      targetPort: 6379
      name: redis
    - port: 6380         # TLS port
      targetPort: 6380
      name: redis-tls
  selector:
    app: redis
  type: NodePort 
EOF
  • Expecting results example: redis server pod is running and service available
$ oc get svc |grep ^redis
NAME    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
redis   NodePort   172.30.56.166   <none>        6379:31290/TCP,6380:32389/TCP   10m

$ oc get pod |grep ^redis-
NAME                     READY   STATUS    RESTARTS   AGE
redis-5dc466fc8b-764hl   1/1     Running   0          10m

2. Certificates preparing

- Create CA, Client and Server Certificates, using Server IP as Common Name (CN): - Create directory `Certs`, `cd Certs`, and run following commands to create server and client certificates, that will be used for test. 
openssl genpkey -algorithm RSA -out ca.key
openssl req -x509 -key ca.key -out ca.crt -days 365 -subj "/CN=172.30.47.231"

openssl genpkey -algorithm RSA -out redis-client.key
openssl req -new -key redis-client.key -out redis-client.csr -subj "/CN=redis-client.example.com"
openssl x509 -req -in redis-client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out redis-client.crt -days 365

openssl genpkey -algorithm RSA -out redis-server.key
openssl req -new -key redis-server.key -out redis-server.csr -subj "/CN=172.30.47.231"
openssl x509 -req -in redis-server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out redis-server.crt -days 365
  • Expected files to be created:
Certs $ ls
ca.crt			redis-client.crt	redis-server.crt
ca.key			redis-client.csr	redis-server.csr
ca.srl			redis-client.key	redis-server.key

3. Update Redis Server with new server certificate

  • Update redis-tls-secret secret, using new created:

    • ca.crt
    • redis-server.crt
    • redis-server.key

  • Restart redis pod

4. Install 3scale

#### 4.1. Create Redis secrets for 3scale
- Below is a script to create the system-redis and backend-redis secrets with dummy client certificates. In the next step, we will replace these dummy certificates with valid client certificates, using the UI for convenience. The valid certificates will be sourced from the files created in the previous section. Additionally, we will update the Redis server URL to reflect the service IP of our Redis server.

You may prepare the secrets in whichever way is most convenient for you.

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata:
  name: system-redis
  namespace: 3scale-test
  labels:
    apimanager.apps.3scale.net/watched-by: system
    app: 3scale-api-management
    threescale_component: system
data:
  SENTINEL_HOSTS: ''
  SENTINEL_ROLE: ''
  REDIS_SSL_CA: ''
  REDIS_SSL_CERT: ''
  REDIS_SSL_KEY: ''
  URL: cmVkaXNzOi8vMTcyLjMwLjU2LjE2Njo2MzgwLzI=
type: Opaque
EOF

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata:
  name: backend-redis
  namespace: 3scale-test
  labels:
    apimanager.apps.3scale.net/watched-by: backend
    app: 3scale-api-management
    threescale_component: backend
data:
  REDIS_STORAGE_URL: cmVkaXNzOi8vMTcyLjMwLjU2LjE2Njo2MzgwLzA=
  REDIS_QUEUES_SENTINEL_HOSTS: ''
  REDIS_STORAGE_SENTINEL_ROLE: ''
  REDIS_SSL_CA: ''
  REDIS_SSL_CERT: ''
  REDIS_SSL_KEY: ''
  REDIS_SSL_QUEUES_CA: ''
  REDIS_SSL_QUEUES_CERT: ''
  REDIS_SSL_QUEUES_KEY: ''
  REDIS_QUEUES_URL: cmVkaXNzOi8vMTcyLjMwLjU2LjE2Njo2MzgwLzE=
  REDIS_QUEUES_SENTINEL_ROLE: ''
  REDIS_STORAGE_SENTINEL_HOSTS: ''
type: Opaque
EOF

  • Update Client Certificates in system-redis and backedn-redis secrets via UI. The following tables are for matching data field names with the certificate files created before:

  • Secret: system-redis

Data field Certificate file name
REDIS_SSL_CA ca.crt
REDIS_SSL_CERT redis-client.crt
REDIS_SSL_KEY redis-client.key
  • Secret: backend-redis
Data field Certificate file name
REDIS_SSL_CA ca.crt
REDIS_SSL_CERT redis-client.crt
REDIS_SSL_KEY redis-client.key
REDIS_SSL_QUEUES_CA ca.crt
REDIS_SSL_QUEUES_CERT redis-client.crt
REDIS_SSL_QUEUES_KEY redis-client.key

Please note:

  • We are using a common CA for both the Redis server and client certificates.
  • We are using the same client certificates for both the Redis system and the backend, including QUEUES.
  • Please don't forget to update the Redis IP in the URLs to match the service IP and CN.
  • Use secure Redis URLs. They should look like rediss://<redis service IP>:6380/0, where rediss indicates a secure connection and port 6380 is used for secure Redis connections. For example:
    • REDIS_QUEUES_URL: rediss://172.30.56.166:6380/1
    • REDIS_STORAGE_URL: rediss://172.30.56.166:6380/0
    • URL: rediss://172.30.56.166:6380/2

4.2. Create s3-credentials secret

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata: 
  name: s3-credentials
  namespace: 3scale-test
data: 
  AWS_ACCESS_KEY_ID: QUtJQVY2SVpYMk9ZQ09OWERFSlkK
  AWS_SECRET_ACCESS_KEY: aU5VbWdZY3hjSDF3azBlUlB0SytmTERHVVMvU0hxM1pKNVBlQy9xYQo=
  AWS_BUCKET: dm1vY2NzZjZxOGxyZWRoYXRyaG9hbW9wZXJhdG9ydGhyZWUtYW9mZwo=
  AWS_REGION: ZXUtd2VzdC0xCg==
type: Opaque
EOF

4.3. Create APIManager CR and Run Operator

Please set wildcardDomain before creation APIManager: 
DOMAIN=$(oc get routes console -n openshift-console -o json | jq -r '.status.ingress[0].routerCanonicalHostname' | sed 's/router-default.//')

cat << EOF | oc create -f -
apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
  name: 3scale
spec:
  system: 
    systemRedisTLSEnabled: true
    fileStorage: 
      simpleStorageService: 
        configurationSecretRef: 
          name: s3-credentials
  backend:
    backendRedisTLSEnabled: true
    queuesRedisTLSEnabled: true    
  wildcardDomain: $DOMAIN
  externalComponents:
    backend:
      redis: true
    system:
      redis: true
      database: true
EOF
  • run Operator to install 3scale: 
    make run

5. Check results

  • Check Environment Variables and Certificates are set in Pods

System pods: system-sidekiq and system-app

  • Login to the pod (oc rsh podname), and run the following commands to verify that the certificate environment variables are defined and the certificate files are populated for the System:
env |grep -E "REDIS_CLIENT_CERT|BACKEND_REDIS_CLIENT_CERT"
env |grep -E "REDIS_CA_FILE|BACKEND_REDIS_CA_FILE"
env |grep -E "REDIS_PRIVATE_KEY|BACKEND_REDIS_PRIVATE_KEY"
env |grep -E "REDIS_SSL|BACKEND_REDIS_SSL"

cat /tls/system-redis/system-redis-ca.crt
cat /tls/system-redis/system-redis-client.crt
cat /tls/system-redis/system-redis-private.key
cat /tls/backend-redis-ca.crt
cat /tls/backend-redis-client.crt
cat /tls/backend-redis-private.key
  • Backend pods:backend-cron, backend-listener, backend-worker
  • Login to the pod (oc rsh podname), and run the following commands to verify that the certificate environment variables are defined and the certificate files are populated for the Backend:
env |grep -E "CONFIG_REDIS_CA_FILE|CONFIG_QUEUES_CA_FILE"  
env |grep -E "CONFIG_REDIS_CERT|CONFIG_QUEUES_CERT"
env |grep -E "CONFIG_REDIS_PRIVATE_KEY|CONFIG_QUEUES_PRIVATE_KEY"
env |grep -E "CONFIG_REDIS_SSL|CONFIG_QUEUES_SSL"

cat /tls/queues/config-queues-ca.crt
cat /tls/queues/config-queues-client.crt
cat /tls/queues/config-queues-private.key
cat /tls/backend-redis-ca.crt
cat /tls/backend-redis-client.crt
cat /tls/backend-redis-private.key

6. Check validation

Test 1

  1. APIManagerCR: Redis TLS is Enabled for system, backend and queues.
  2. backend-redis and system-redis secrets are missing required TLS fields
  • Expected results:
    • installation is not progressing
    • backend-redis secret validation errors in operator log:
2025-02-10T07:54:15+02:00	ERROR	Reconciler error	{"controller": "apimanager", "controllerGroup": "apps.3scale.net", "controllerKind": "APIManager", "APIManager": {"name":"3scale","namespace":"3scale-test"}, "namespace": "3scale-test", "name": "3scale", "reconcileID": "27ade290-b64f-4b40-b99f-8fc4f90587b7", "error": "validation errors for Redis TLS configuration in 'backend-redis' secret: 'backendRedisTLSEnabled: true' is set in apimanager. Secret validation errors: [Secret field 'REDIS_SSL_CA' is required in secret 'backend-redis' Secret field 'REDIS_SSL_CERT' is required in secret 'backend-redis' Secret field 'REDIS_SSL_KEY' is required in secret 'backend-redis']\n'queuesRedisTLSEnabled: true' is set in apimanager. Secret validation errors: [Secret field 'REDIS_SSL_QUEUES_CA' is required in secret 'backend-redis' Secret field 'REDIS_SSL_QUEUES_CERT' is required in secret 'backend-redis' Secret field 'REDIS_SSL_QUEUES_KEY' is required in secret 'backend-redis']"}

Test 2

  1. APIManagerCR: Redis TLS is Enabled for system, backend and queues.
  2. backend-redis secret - TLS fields are empty
  3. system-redis secrets is missing required TLS fields
  • Expected results:
    • installation is not progressing
    • system-redis secret - validation errors in operator log - notification that fields are required
2025-02-10T08:43:20+02:00	ERROR	Reconciler error	{"controller": "apimanager", "controllerGroup": "apps.3scale.net", "controllerKind": "APIManager", "APIManager": {"name":"3scale","namespace":"3scale-test"}, "namespace": "3scale-test", "name": "3scale", "reconcileID": "b710a16d-e567-4e9a-b553-4347e83ea744", "error": "validation errors for Redis TLS configuration in 'system-redis' secret: Secret field 'REDIS_SSL_CA' is required in secret 'system-redis'\nSecret field 'REDIS_SSL_CERT' is required in secret 'system-redis'\nSecret field 'REDIS_SSL_KEY' is required in secret 'system-redis'"}

7. Dedicated Redis Servers for System, Backend and Queue

In this section we will create 3 redis servers - dedicated for System, Backend and Queues.
  • preparation

cd 3scale-operator
make install
export NAMESPACE=3scale-test
oc new-project $NAMESPACE
oc project $NAMESPACE
make download
make cluster/create/system-postgres
export PREFLIGHT_CHECKS_BYPASS=true

  • We are following the same approach and using the same scripts as in previous sections, with the following important points to note:

    • Use unique and distinct ports for each Redis server.
    • Use dedicated certificates for each Redis server to ensure proper TLS configuration.

  • Below, we provide the details to create redis2 and redis3 for the Backend and Queues.

    • For the System configuration, you can refer to Section 1 of this document.

  • Create redis server for Backend - redis2


cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata:
  name: redis-tls-secret-2
  namespace: 3scale-test
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lVYS9FaGY1UDR3WWVnYVZhaGpqT0R6ME1QVTJnd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF3d09NVGN5TGpNd0xqSXdOeTR4T1RFd0hoY05NalF4TVRJME1EZ3lNakF6V2hjTgpNalV4TVRJME1EZ3lNakF6V2pBWk1SY3dGUVlEVlFRRERBNHhOekl1TXpBdU1qQTNMakU1TVRDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUl0eld6OHcrTDBOYXAzN0FIcXR4Rjh6OEdXRFROUnoKOUxVa0JYcXU1SkZhc1lxejYrZ2k3S3dsWUlTeEFwVzd4NXprWDZ4bjBCeGxsMXVNdzExZHVyeDJNUjhlYXFxQQpxbElFVzZwTU1kTGNndkhJUit0TGhLRFVvaVpsajdKRkk1MGYvYjhUaXRKVTByY1dhYmd4SVA4QnlINUdxSkR2CmtFZGgvNC9SbTlYeWU4SmhuMEVqRzdPUWxTM1MvOGJHSGpoV2Zjdys2QnJidDI5TE1aTzdJRmFlRVh4azRaQjAKS1VNODJ0TFhxV1VwaUdxQUhuWncrZUtBSlZDOFBzUzB5NU9aTGt4TE9GUlJnajJRYkF3TlZ1emxBN0VMQ0U4dApyVkx3OWV3aEhsVkhUcVZ0UXYwNWZVb2ZqOURkb3Yxd3F2OG1uZXJ3OHBnRVR3Yi9RemY1VGRjQ0F3RUFBYU5UCk1GRXdIUVlEVlIwT0JCWUVGRGpocmlHaDZxbUZVS2Q0OG14UERUWi9CYThBTUI4R0ExVWRJd1FZTUJhQUZEamgKcmlHaDZxbUZVS2Q0OG14UERUWi9CYThBTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFDdCtPNTZnanZVTUo5MFdaekFFcEFLcTNFTDY3c1I4NXlYam5EeXZML1lRRjZZSzRhUmVaZ1JICms5WCs2cDNyd2tsQmF6MzFFTFNFSGlPUkVTWDJzNW9WSVlId01SNUpsOE5VZTB6NGVBZlNNcTNtQmFuYWxneWYKQktROHZRc1UxTGVJQkpGRVJ5dG5aVE03VThFbFVsUkZSZzAzVUcyVG5ibGNxalFRa1lTMmZYT3M1czBlMHI2eQpaektXVWZhemtzTlkzSHlUcUxRTDlEZUM5STd6R0s2c2E5dHhueTFOM0I5bWJHZ1l3dlFZNDlvTm1kem9oWW16CkxmVVZJeERSWUcwNnI1UldoTTVtZDhZNmJUSGo1TExHT1A2YlEvYWJteXZxbFpuVnZjWTZaandKUzhNWVJJSk8KMmxrczl3cXRjeENla3VJZFBrOUNsNlVBZ3E5TkJoVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  redis-server.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURBakNDQWVxZ0F3SUJBZ0lVYUIyZGNLRXJuRHM0MEVqMitwT210bmpRQytVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF3d09NVGN5TGpNd0xqSXdOeTR4T1RFd0hoY05NalF4TVRJME1EZ3lNakF6V2hjTgpNalV4TVRJME1EZ3lNakF6V2pBWk1SY3dGUVlEVlFRRERBNHhOekl1TXpBdU1qQTNMakU1TVRDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQVAxN2tGaHBRdDhpcUlRWEJOUEozbHBrOFFGUWJnUncKc2VlQkNoa1RTYTd5S0hvQS9VTHVHY3QzZHVubWZPeWdNeEhueW9LUmR0NDVnWWlDOGNnQjE4OWRDRHo2cy84ZAp2TUdoUFUrblhkOWxyVHNtYXpLNW9McnVETmI3TXhBVDFVdjNIMXkvbFErY2tPTkttLzVhWndoNnFxL25XVFZtCkdOZUVGcDJyNUczTkVXcTJXWm1KM0RCc3pMeUdFciszdUlJbFVsaUk2bXRZUGRRd1QwR3RqTDJpVldRVUd2Q2sKYWpHclNJQUVETVFCdFNYeE1yOTlzS2VhS21iVDJ1L09BOEJjYVlqbXdZVnN4VDIzczR2UFltaUhjTFVjVFE4MApTblBLSW03NDJrTzhLZE5Id1hjN1BpejhUTVB0WTU1QkU5ZlBIaGdueVNBMVZUeWNQR1cyZEMwQ0F3RUFBYU5DCk1FQXdIUVlEVlIwT0JCWUVGT3BwbncveVd2dFo2L0llVzhUUlhXeG1uMUtXTUI4R0ExVWRJd1FZTUJhQUZEamgKcmlHaDZxbUZVS2Q0OG14UERUWi9CYThBTUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFCWnhOZGpZR1Y4M1RBUwp6amMyNjZnc3gwRTB0cTRVUnRGWHdFeXZTYkxvY3BmK1V2OCtmTFBYZGF0WG1lRm5MdlJZbXg0VGR1UlltMFBYCk1BRUpqcEE3UXdlM1NsSmd5R2loNk1PVVdIRFdNZnhQZWJQWUh5ZFQ3TXBKVVJNWkxjZ0tvWlVENWNKdDc1bVIKMk51Z3BvdDdxQnYybzZGdmI5cmpNYSt1WFdJUEdSekhidjJ6eVFSR1J3SUpZcitKaDNWbXcyRSt0Vit0VEUwQQovWlBxOTIxYnlWYkREZHI2aDMyUnRRN1NJUnAwLzBnWFZablY3NTB2Rm9Ub2xOK2wwLzJOeEVCWmZNSzVFcEdkCkViSFFvN2dETHg0REljZlhqZVlxUkFuYjNLbzhXK0ZrQS8wQUdQS2NSN2RIL2daVnFHMngrRk5QazQ4YzBDQkwKR0R2RjlRYncKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  redis-server.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRRDllNUJZYVVMZklxaUUKRndUVHlkNWFaUEVCVUc0RWNMSG5nUW9aRTBtdThpaDZBUDFDN2huTGQzYnA1bnpzb0RNUjU4cUNrWGJlT1lHSQpndkhJQWRmUFhRZzgrclAvSGJ6Qm9UMVBwMTNmWmEwN0ptc3l1YUM2N2d6Vyt6TVFFOVZMOXg5Y3Y1VVBuSkRqClNwditXbWNJZXFxdjUxazFaaGpYaEJhZHErUnR6UkZxdGxtWmlkd3diTXk4aGhLL3Q3aUNKVkpZaU9wcldEM1UKTUU5QnJZeTlvbFZrRkJyd3BHb3hxMGlBQkF6RUFiVWw4VEsvZmJDbm1pcG0wOXJ2emdQQVhHbUk1c0dGYk1VOQp0N09MejJKb2gzQzFIRTBQTkVwenlpSnUrTnBEdkNuVFI4RjNPejRzL0V6RDdXT2VRUlBYeng0WUo4a2dOVlU4Cm5EeGx0blF0QWdNQkFBRUNnZ0VBRG5vaktWbUJyenJNZ3hiSmVNc2J2dS9xNzlkSElVdktiVjFxVlRwTHlBa2UKbExFL3hiWFJsVlJTWDFPQnFRWVJSS0dIYUdPa2RWYTFkalY4VjU3N1UyV04xZVcvcC85cnkyZEpHQ2FIN3YxZwpvbk0wUmlaaDdxc3Y0b3RnUkRmTnc5UHVYNTYxaGJtOGNLN1BML3k3eTdrdHpIUWJIVGlpakpTSHNpT2lIVDh1CitWc3oySzZmSnhHNUE2SE03bldvdlBuK1VydFd4OGd4OEJNQ1FOa3dUVnQvVmxzcE1wcUo1czdqeUNncGFQN2MKV0wrWnh2K2xUcFUwcnpJUTFZMWYvL0RSeUI3OVEycUl0dVhEdkJnNnI5ckExbytSV0JuNnA3WVA0T21SUE4vZQpiMVZZcG14c1BBL2ppQkhsRVFPZVJWbEVSYzhJSGVib2IxZFZ1Rm15b1FLQmdRRC9LeFlvVzNxaGNmUEYya2EvCmliZTkvKzlyQzNlTkdHUGwyaDVCeDJwK3k4dnlQb3NabmZ5STRKRzNtcXpVS1luSHFPVzhpTVpnR2ZOUlRNdVQKeVFSQitsVlFldHpOcDJUZCt4dWlCMUJmWHIyanowcVZvV3F4dFhrMklRWXFJN2NMSGZVZ2dGeUtsL2lRRGZlOQpwd0YwZ3c5Nm1vT012VWp1S1FmZjh2RTYxUUtCZ1FEK1R4SWZtc1VyY2VFdmpkeUdhNllweTBZcEx2OVhnSk0rCkxIc1ZmVTgzNXNWeVJaUDZrV1EwSGwyVCtHMDBqcStsU20vQ296cWNZeE5ObmtGMEhOdEw2SlJJWTBZZEExSFUKSmc1eTY3U2w0U0hHeWdYNTdrZEE4ajhyS1I4UXpmQ0RHT3Q5d2NwNjZ4MVlMSWEwRDYvV2h3elFOeXl3QzdnTwoxS3dvZlhyUCtRS0JnRzU4WUk2KzlYMWNVdnBUaGhpL2IvRDBGZDNhekR3cTJHNlpJRXJKSndLYUNjZnRidHQ3CnZmSWlrdFhXUW9sbkp3SnR6blB4SVR4UllEck9yc05oNGRjVHBzYy9POFpNZWU5b0lGSHJLdER3dTlwbkVsdHgKMWpuMll2S2VJQVkxQ3Jma2s5UXI0R1llWVlFMm14UGljVTNheGVRSGJYaU9LVHIrUnl1Z0RQVzFBb0dBZm5vdwoxMHNRR0tWUWkyZ1FiMElHcCs2Uy9GU0ZaYTFxalpkdHQ2aFV4OGFjR0ZNR1g2NERtZkFvTmpsdGhxQVlOeXFvCkhyTXpxU2VWS0JzM0RscHpybk1EbkdUVE1BYkFvYlF6cDNBV3JoRWp6VXdZWU03aTNTZ2R4b2R6RGRaK2NaVHAKT2VneG5hUmxPYjhiVjE0ZDQ2SFMrNU1WUkpEdmYyRENKbmtScFhFQ2dZRUFnVFB3MlVaSkJRVVdGc0NVRjczdgpGcWZSbzdZYmtweDAxcFlBcmV0ak41TTR6ZHZRUlZPQlFmVGRJTU9SbFJ4OVBnRitYL3UrT0szUzB1aS8xdWlECjhYU2QvMXFjS3ZOdUJ6ZnZNdTVGeVdKRHNCUGJaUHZuWnEzdTM5WktaYTNXNG92UE9VWlFERGlCTHNhM21CekwKcFdYNWFyVHNTazRSSDExQ28zRTN0akU9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
type: Opaque
EOF


cat << EOF | oc create -f -
apiVersion: v1
data:
  redis.conf: |+
    # redis.conf
    bind 0.0.0.0
    protected-mode no
    port 6381
    tls-port 6382
    tls-cert-file /etc/redis/certs/redis-server.crt
    tls-key-file /etc/redis/certs/redis-server.key
    tls-ca-cert-file /etc/redis/certs/ca.crt
    tls-auth-clients yes
    stop-writes-on-bgsave-error no
    save ""
kind: ConfigMap
metadata:
  name: redis-config-redis2
EOF


cat << EOF | oc create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis2
  template:
    metadata:
      labels:
        app: redis2
    spec:
      containers:
      - name: redis
        image: quay.io/fedora/redis-6
        ports:
        - containerPort: 6381
        - containerPort: 6382
        volumeMounts:
        - name: redis-config-volume
          mountPath: /etc/redis/redis.conf
          subPath: redis.conf
        - name: redis-tls-volume
          mountPath: /etc/redis/certs
          readOnly: true
        command: ["/bin/sh", "-c", "redis-server /etc/redis/redis.conf"]
      volumes:
      - name: redis-config-volume
        configMap:
          name: redis-config-redis2
      - name: redis-tls-volume
        secret:
          secretName: redis-tls-secret-2
EOF

cat << EOF | oc create -f -
apiVersion: v1
kind: Service
metadata:
  name: redis2
spec:
  ports:
    - port: 6381         # Non-TLS (unencrypted) port
      targetPort: 6381
      name: redis
    - port: 6382         # TLS port
      targetPort: 6382
      name: redis-tls
  selector:
    app: redis2
  type: NodePort 
EOF
  • Create redis server - for Queue - redis3 

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata:
  name: redis-tls-secret-3
  namespace: 3scale-test
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lVYS9FaGY1UDR3WWVnYVZhaGpqT0R6ME1QVTJnd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF3d09NVGN5TGpNd0xqSXdOeTR4T1RFd0hoY05NalF4TVRJME1EZ3lNakF6V2hjTgpNalV4TVRJME1EZ3lNakF6V2pBWk1SY3dGUVlEVlFRRERBNHhOekl1TXpBdU1qQTNMakU1TVRDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUl0eld6OHcrTDBOYXAzN0FIcXR4Rjh6OEdXRFROUnoKOUxVa0JYcXU1SkZhc1lxejYrZ2k3S3dsWUlTeEFwVzd4NXprWDZ4bjBCeGxsMXVNdzExZHVyeDJNUjhlYXFxQQpxbElFVzZwTU1kTGNndkhJUit0TGhLRFVvaVpsajdKRkk1MGYvYjhUaXRKVTByY1dhYmd4SVA4QnlINUdxSkR2CmtFZGgvNC9SbTlYeWU4SmhuMEVqRzdPUWxTM1MvOGJHSGpoV2Zjdys2QnJidDI5TE1aTzdJRmFlRVh4azRaQjAKS1VNODJ0TFhxV1VwaUdxQUhuWncrZUtBSlZDOFBzUzB5NU9aTGt4TE9GUlJnajJRYkF3TlZ1emxBN0VMQ0U4dApyVkx3OWV3aEhsVkhUcVZ0UXYwNWZVb2ZqOURkb3Yxd3F2OG1uZXJ3OHBnRVR3Yi9RemY1VGRjQ0F3RUFBYU5UCk1GRXdIUVlEVlIwT0JCWUVGRGpocmlHaDZxbUZVS2Q0OG14UERUWi9CYThBTUI4R0ExVWRJd1FZTUJhQUZEamgKcmlHaDZxbUZVS2Q0OG14UERUWi9CYThBTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFDdCtPNTZnanZVTUo5MFdaekFFcEFLcTNFTDY3c1I4NXlYam5EeXZML1lRRjZZSzRhUmVaZ1JICms5WCs2cDNyd2tsQmF6MzFFTFNFSGlPUkVTWDJzNW9WSVlId01SNUpsOE5VZTB6NGVBZlNNcTNtQmFuYWxneWYKQktROHZRc1UxTGVJQkpGRVJ5dG5aVE03VThFbFVsUkZSZzAzVUcyVG5ibGNxalFRa1lTMmZYT3M1czBlMHI2eQpaektXVWZhemtzTlkzSHlUcUxRTDlEZUM5STd6R0s2c2E5dHhueTFOM0I5bWJHZ1l3dlFZNDlvTm1kem9oWW16CkxmVVZJeERSWUcwNnI1UldoTTVtZDhZNmJUSGo1TExHT1A2YlEvYWJteXZxbFpuVnZjWTZaandKUzhNWVJJSk8KMmxrczl3cXRjeENla3VJZFBrOUNsNlVBZ3E5TkJoVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  redis-server.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURBakNDQWVxZ0F3SUJBZ0lVYUIyZGNLRXJuRHM0MEVqMitwT210bmpRQytVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF3d09NVGN5TGpNd0xqSXdOeTR4T1RFd0hoY05NalF4TVRJME1EZ3lNakF6V2hjTgpNalV4TVRJME1EZ3lNakF6V2pBWk1SY3dGUVlEVlFRRERBNHhOekl1TXpBdU1qQTNMakU1TVRDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQVAxN2tGaHBRdDhpcUlRWEJOUEozbHBrOFFGUWJnUncKc2VlQkNoa1RTYTd5S0hvQS9VTHVHY3QzZHVubWZPeWdNeEhueW9LUmR0NDVnWWlDOGNnQjE4OWRDRHo2cy84ZAp2TUdoUFUrblhkOWxyVHNtYXpLNW9McnVETmI3TXhBVDFVdjNIMXkvbFErY2tPTkttLzVhWndoNnFxL25XVFZtCkdOZUVGcDJyNUczTkVXcTJXWm1KM0RCc3pMeUdFciszdUlJbFVsaUk2bXRZUGRRd1QwR3RqTDJpVldRVUd2Q2sKYWpHclNJQUVETVFCdFNYeE1yOTlzS2VhS21iVDJ1L09BOEJjYVlqbXdZVnN4VDIzczR2UFltaUhjTFVjVFE4MApTblBLSW03NDJrTzhLZE5Id1hjN1BpejhUTVB0WTU1QkU5ZlBIaGdueVNBMVZUeWNQR1cyZEMwQ0F3RUFBYU5DCk1FQXdIUVlEVlIwT0JCWUVGT3BwbncveVd2dFo2L0llVzhUUlhXeG1uMUtXTUI4R0ExVWRJd1FZTUJhQUZEamgKcmlHaDZxbUZVS2Q0OG14UERUWi9CYThBTUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFCWnhOZGpZR1Y4M1RBUwp6amMyNjZnc3gwRTB0cTRVUnRGWHdFeXZTYkxvY3BmK1V2OCtmTFBYZGF0WG1lRm5MdlJZbXg0VGR1UlltMFBYCk1BRUpqcEE3UXdlM1NsSmd5R2loNk1PVVdIRFdNZnhQZWJQWUh5ZFQ3TXBKVVJNWkxjZ0tvWlVENWNKdDc1bVIKMk51Z3BvdDdxQnYybzZGdmI5cmpNYSt1WFdJUEdSekhidjJ6eVFSR1J3SUpZcitKaDNWbXcyRSt0Vit0VEUwQQovWlBxOTIxYnlWYkREZHI2aDMyUnRRN1NJUnAwLzBnWFZablY3NTB2Rm9Ub2xOK2wwLzJOeEVCWmZNSzVFcEdkCkViSFFvN2dETHg0REljZlhqZVlxUkFuYjNLbzhXK0ZrQS8wQUdQS2NSN2RIL2daVnFHMngrRk5QazQ4YzBDQkwKR0R2RjlRYncKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  redis-server.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRRDllNUJZYVVMZklxaUUKRndUVHlkNWFaUEVCVUc0RWNMSG5nUW9aRTBtdThpaDZBUDFDN2huTGQzYnA1bnpzb0RNUjU4cUNrWGJlT1lHSQpndkhJQWRmUFhRZzgrclAvSGJ6Qm9UMVBwMTNmWmEwN0ptc3l1YUM2N2d6Vyt6TVFFOVZMOXg5Y3Y1VVBuSkRqClNwditXbWNJZXFxdjUxazFaaGpYaEJhZHErUnR6UkZxdGxtWmlkd3diTXk4aGhLL3Q3aUNKVkpZaU9wcldEM1UKTUU5QnJZeTlvbFZrRkJyd3BHb3hxMGlBQkF6RUFiVWw4VEsvZmJDbm1pcG0wOXJ2emdQQVhHbUk1c0dGYk1VOQp0N09MejJKb2gzQzFIRTBQTkVwenlpSnUrTnBEdkNuVFI4RjNPejRzL0V6RDdXT2VRUlBYeng0WUo4a2dOVlU4Cm5EeGx0blF0QWdNQkFBRUNnZ0VBRG5vaktWbUJyenJNZ3hiSmVNc2J2dS9xNzlkSElVdktiVjFxVlRwTHlBa2UKbExFL3hiWFJsVlJTWDFPQnFRWVJSS0dIYUdPa2RWYTFkalY4VjU3N1UyV04xZVcvcC85cnkyZEpHQ2FIN3YxZwpvbk0wUmlaaDdxc3Y0b3RnUkRmTnc5UHVYNTYxaGJtOGNLN1BML3k3eTdrdHpIUWJIVGlpakpTSHNpT2lIVDh1CitWc3oySzZmSnhHNUE2SE03bldvdlBuK1VydFd4OGd4OEJNQ1FOa3dUVnQvVmxzcE1wcUo1czdqeUNncGFQN2MKV0wrWnh2K2xUcFUwcnpJUTFZMWYvL0RSeUI3OVEycUl0dVhEdkJnNnI5ckExbytSV0JuNnA3WVA0T21SUE4vZQpiMVZZcG14c1BBL2ppQkhsRVFPZVJWbEVSYzhJSGVib2IxZFZ1Rm15b1FLQmdRRC9LeFlvVzNxaGNmUEYya2EvCmliZTkvKzlyQzNlTkdHUGwyaDVCeDJwK3k4dnlQb3NabmZ5STRKRzNtcXpVS1luSHFPVzhpTVpnR2ZOUlRNdVQKeVFSQitsVlFldHpOcDJUZCt4dWlCMUJmWHIyanowcVZvV3F4dFhrMklRWXFJN2NMSGZVZ2dGeUtsL2lRRGZlOQpwd0YwZ3c5Nm1vT012VWp1S1FmZjh2RTYxUUtCZ1FEK1R4SWZtc1VyY2VFdmpkeUdhNllweTBZcEx2OVhnSk0rCkxIc1ZmVTgzNXNWeVJaUDZrV1EwSGwyVCtHMDBqcStsU20vQ296cWNZeE5ObmtGMEhOdEw2SlJJWTBZZEExSFUKSmc1eTY3U2w0U0hHeWdYNTdrZEE4ajhyS1I4UXpmQ0RHT3Q5d2NwNjZ4MVlMSWEwRDYvV2h3elFOeXl3QzdnTwoxS3dvZlhyUCtRS0JnRzU4WUk2KzlYMWNVdnBUaGhpL2IvRDBGZDNhekR3cTJHNlpJRXJKSndLYUNjZnRidHQ3CnZmSWlrdFhXUW9sbkp3SnR6blB4SVR4UllEck9yc05oNGRjVHBzYy9POFpNZWU5b0lGSHJLdER3dTlwbkVsdHgKMWpuMll2S2VJQVkxQ3Jma2s5UXI0R1llWVlFMm14UGljVTNheGVRSGJYaU9LVHIrUnl1Z0RQVzFBb0dBZm5vdwoxMHNRR0tWUWkyZ1FiMElHcCs2Uy9GU0ZaYTFxalpkdHQ2aFV4OGFjR0ZNR1g2NERtZkFvTmpsdGhxQVlOeXFvCkhyTXpxU2VWS0JzM0RscHpybk1EbkdUVE1BYkFvYlF6cDNBV3JoRWp6VXdZWU03aTNTZ2R4b2R6RGRaK2NaVHAKT2VneG5hUmxPYjhiVjE0ZDQ2SFMrNU1WUkpEdmYyRENKbmtScFhFQ2dZRUFnVFB3MlVaSkJRVVdGc0NVRjczdgpGcWZSbzdZYmtweDAxcFlBcmV0ak41TTR6ZHZRUlZPQlFmVGRJTU9SbFJ4OVBnRitYL3UrT0szUzB1aS8xdWlECjhYU2QvMXFjS3ZOdUJ6ZnZNdTVGeVdKRHNCUGJaUHZuWnEzdTM5WktaYTNXNG92UE9VWlFERGlCTHNhM21CekwKcFdYNWFyVHNTazRSSDExQ28zRTN0akU9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
type: Opaque
EOF


cat << EOF | oc create -f -
apiVersion: v1
data:
  redis.conf: |+
    # redis.conf
    bind 0.0.0.0
    protected-mode no
    port 6383
    tls-port 6384
    tls-cert-file /etc/redis/certs/redis-server.crt
    tls-key-file /etc/redis/certs/redis-server.key
    tls-ca-cert-file /etc/redis/certs/ca.crt
    tls-auth-clients yes
    stop-writes-on-bgsave-error no
    save ""
kind: ConfigMap
metadata:
  name: redis-config-redis3
EOF


cat << EOF | oc create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis3
  template:
    metadata:
      labels:
        app: redis3
    spec:
      containers:
      - name: redis
        image: quay.io/fedora/redis-6
        ports:
        - containerPort: 6383
        - containerPort: 6384
        volumeMounts:
        - name: redis-config-volume
          mountPath: /etc/redis/redis.conf
          subPath: redis.conf
        - name: redis-tls-volume
          mountPath: /etc/redis/certs
          readOnly: true
        command: ["/bin/sh", "-c", "redis-server /etc/redis/redis.conf"]
      volumes:
      - name: redis-config-volume
        configMap:
          name: redis-config-redis3
      - name: redis-tls-volume
        secret:
          secretName: redis-tls-secret-3
EOF


cat << EOF | oc create -f -
apiVersion: v1
kind: Service
metadata:
  name: redis3
spec:
  ports:
    - port: 6383         # Non-TLS (unencrypted) port
      targetPort: 6383
      name: redis
    - port: 6384         # TLS port
      targetPort: 6384
      name: redis-tls
  selector:
    app: redis3
  type: NodePort 
EOF
  • Services created:

oc get svc |grep ^redis
redis                NodePort    172.30.154.221   <none>        6379:30918/TCP,6380:31415/TCP   92m
redis2               NodePort    172.30.225.37    <none>        6381:30421/TCP,6382:31600/TCP   10m
redis3               NodePort    172.30.100.112   <none>        6383:32766/TCP,6384:31825/TCP   9s
  • Use Services redis-2 and redis-3 IPs to create Certs for Backend and Queues
    • using same openssl commands for certs creation in section 2

openssl genpkey -algorithm RSA -out ca.key
openssl req -x509 -key ca.key -out ca.crt -days 365 -subj "/CN=172.30.100.112"

openssl genpkey -algorithm RSA -out redis-client.key
openssl req -new -key redis-client.key -out redis-client.csr -subj "/CN=redis-client.example.com"
openssl x509 -req -in redis-client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out redis-client.crt -days 365

openssl genpkey -algorithm RSA -out redis-server.key
openssl req -new -key redis-server.key -out redis-server.csr -subj "/CN=172.30.100.112"
openssl x509 -req -in redis-server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out redis-server.crt -days 365
  • Update redis-tls-secret-2 and redis-tls-secret-3 secrets to use new Server Certs
  • Restart redis-2 and redis-3 Pods
  • Update backend-redis secret, to use new Client Certs and new services IPs
    • check URLs and Ports in backedn-secret, that it points to corresponding Redis services

oc get secret backend-redis -oyaml | grep URL
  REDIS_QUEUES_URL: cmVkaXNzOi8vMTcyLjMwLjEwMC4xMTI6NjM4NC8x
  REDIS_STORAGE_URL: cmVkaXNzOi8vMTcyLjMwLjIyNS4zNzo2MzgyLzA=

echo cmVkaXNzOi8vMTcyLjMwLjEwMC4xMTI6NjM4NC8x |base64 -d
rediss://172.30.100.112:6384/1                                                                                                         

echo cmVkaXNzOi8vMTcyLjMwLjIyNS4zNzo2MzgyLzA= |base64 -d
rediss://172.30.225.37:6382/0

  • Create apimanager and start 3scale-operator:

    • see section 4.3

  • Check installation, see that all pods are up.


oc get deploy |grep ^redis
redis                1/1     1            1           4h13m
redis2               1/1     1            1           35m
redis3               1/1     1            1           25m



oc get pod
NAME                                     READY   STATUS      RESTARTS      AGE
apicast-production-5797bc85b5-hlj8x      1/1     Running     0             62m
apicast-staging-5bb9d8c587-p76qm         1/1     Running     0             62m
backend-cron-69f96c89c9-rdzp2            1/1     Running     0             38s
backend-listener-6bf9d9f648-2hgh5        1/1     Running     0             63m
backend-listener-7846749798-sgjfq        0/1     Running     0             38s
backend-worker-5f8f944d6f-rs82r          1/1     Running     0             38s
redis-5dc466fc8b-vlfrf                   1/1     Running     0             3h15m
redis2-b5789848f-d6b5d                   1/1     Running     0             5m54s
redis3-658db895f8-zcjst                  1/1     Running     0             5m54s
system-app-766cb7cbbb-jjlk8              3/3     Running     0             62m
system-app-post-w9fzw                    0/1     Completed   0             60m
system-app-pre-r9crh                     0/1     Completed   0             62m
system-memcache-b6565f76b-tsv9w          1/1     Running     0             63m
system-postgresql-f45b75766-2msbt        1/1     Running     0             4h44m
system-searchd-669dc7599c-sk57n          1/1     Running     0             63m
system-searchd-manticore-reindex-swlts   0/1     Completed   0             62m
system-sidekiq-5b8d747d8c-hbfm5          1/1     Running     4 (59m ago)   62m
throwaway-redis                          1/1     Running     0             177m
zync-77555479f-8298f                     1/1     Running     0             62m
zync-database-7c446ccdf-wpsq9            1/1     Running     0             62m
zync-que-844ff6987b-pkh4p                1/1     Running     2 (62m ago)   62m

valerymo avatar Nov 25 '24 13:11 valerymo

There is a new functionality added to resync-routes when zync is enabled. This causes a job to be created, resync-route. The job keeps failing due to:

Failed to load CA Certificate or CA Path
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:37:in `initialize'
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:24:in `new'
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:24:in `ssl_context'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client/config.rb:125:in `ssl_context'
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:134:in `connect'
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:50:in `initialize'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:746:in `new'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:746:in `block in connect'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client/middlewares.rb:12:in `connect'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:745:in `connect'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:732:in `raw_connection'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:697:in `ensure_connected'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:292:in `call_v'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis/client.rb:90:in `call_v'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis.rb:152:in `block in send_command'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis.rb:151:in `synchronize'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis.rb:151:in `send_command'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis/commands/connection.rb:21:in `ping'
/opt/system/lib/tasks/boot.rake:26:in `block (3 levels) in <top (required)>'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis.rb:95:in `with'
/opt/system/app/lib/system/redis_pool.rb:26:in `public_send'
/opt/system/app/lib/system/redis_pool.rb:26:in `block in method_missing'

MStokluska avatar Jan 23 '25 12:01 MStokluska

The secrets when TLS is enabled are annotated with: apimanager.apps.3scale.net/watched-by: system and apimanager.apps.3scale.net/watched-by: backend This doesn't work.

To enable watched-by you need apimanager.apps.3scale.net/watched-by: apimanager

MStokluska avatar Jan 23 '25 12:01 MStokluska

In backend-redis, we have redis queues and redis storage entries - think we could maintain that? Or does it have to be now, redis queues and redis?

MStokluska avatar Jan 23 '25 12:01 MStokluska

In backend-redis, we have redis queues and redis storage entries - think we could maintain that? Or does it have to be now, redis queues and redis?

It's in requirements, as in Jira, if I understand the question : Backend: CONFIG_REDIS_CA_FILE and CONFIG_QUEUES_CA_FILE for the CA certificate CONFIG_REDIS_CERT and CONFIG_QUEUES_CERT for the client certificate CONFIG_REDIS_PRIVATE_KEY and CONFIG_QUEUES_PRIVATE_KEY for the client key CONFIG_REDIS_SSL and CONFIG_QUEUES_SSL must be set to true when any of the above is present

valerymo avatar Jan 23 '25 13:01 valerymo

In backend-redis, we have redis queues and redis storage entries - think we could maintain that? Or does it have to be now, redis queues and redis?

It's in requirements, as in Jira, if I understand the question : Backend: CONFIG_REDIS_CA_FILE and CONFIG_QUEUES_CA_FILE for the CA certificate CONFIG_REDIS_CERT and CONFIG_QUEUES_CERT for the client certificate CONFIG_REDIS_PRIVATE_KEY and CONFIG_QUEUES_PRIVATE_KEY for the client key CONFIG_REDIS_SSL and CONFIG_QUEUES_SSL must be set to true when any of the above is present

I think I would change that to be clear and follow what we have, alternatively, bring it up in jira please

MStokluska avatar Jan 23 '25 13:01 MStokluska

When enabling TLS with wrong certs (I've tried with storage) what happens is that although the authentication in backend listener fails, the products can be created and promoted to stage/prod. Auth fails on api requests. Then, when fixing up the certs, the requests are still failing. I believe this requires a redis re-sync?

I guess, porta does some validation right, so maybe an idea here would be to restart system-app pod on every update of the TLS secrets to perform the validation (and hopefully catch the incorrect certs and block api access?)

WDYT?

MStokluska avatar Jan 23 '25 13:01 MStokluska

The secrets when TLS is enabled are annotated with: apimanager.apps.3scale.net/watched-by: system and apimanager.apps.3scale.net/watched-by: backend This doesn't work.

To enable watched-by you need apimanager.apps.3scale.net/watched-by: apimanager

@MStokluska It's working, I tested watched-by in my PR. The reason why it's working is here - checked only key of lable, not value. But Secret itself it's created by end-user, so actually nothing change in code, as I think.

valerymo avatar Jan 23 '25 13:01 valerymo

When enabling TLS with wrong certs (I've tried with storage) what happens is that although the authentication in backend listener fails, the products can be created and promoted to stage/prod. Auth fails on api requests. Then, when fixing up the certs, the requests are still failing. I believe this requires a redis re-sync?

I guess, porta does some validation right, so maybe an idea here would be to restart system-app pod on every update of the TLS secrets to perform the validation (and hopefully catch the incorrect certs and block api access?)

WDYT?

@MStokluska , all backend pods (listener, worker, cron) and system (app and sidekiq) are restarting if any change in secrets system-redis and backend-redis (that hold certificates), as whatched-by is available on backend-redis and system-redis

valerymo avatar Jan 23 '25 13:01 valerymo

In backend-redis, we have redis queues and redis storage entries - think we could maintain that? Or does it have to be now, redis queues and redis?

It's in requirements, as in Jira, if I understand the question : Backend: CONFIG_REDIS_CA_FILE and CONFIG_QUEUES_CA_FILE for the CA certificate CONFIG_REDIS_CERT and CONFIG_QUEUES_CERT for the client certificate CONFIG_REDIS_PRIVATE_KEY and CONFIG_QUEUES_PRIVATE_KEY for the client key CONFIG_REDIS_SSL and CONFIG_QUEUES_SSL must be set to true when any of the above is present

I think I would change that to be clear and follow what we have, alternatively, bring it up in jira please

@MStokluska , will discuss it. Thank you for comments

valerymo avatar Jan 23 '25 13:01 valerymo

When enabling TLS with wrong certs (I've tried with storage) what happens is that although the authentication in backend listener fails, the products can be created and promoted to stage/prod. Auth fails on api requests. Then, when fixing up the certs, the requests are still failing. I believe this requires a redis re-sync? I guess, porta does some validation right, so maybe an idea here would be to restart system-app pod on every update of the TLS secrets to perform the validation (and hopefully catch the incorrect certs and block api access?) WDYT?

@MStokluska , all backend pods (listener, worker, cron) and system (app and sidekiq) are restarting if any change in secrets system-redis and backend-redis (that hold certificates), as whatched-by is available on backend-redis and system-redis

In your current implementation, watched-by doesn't work correctly. When I've fixed it, watched-by on backend-redis doesn't restart system pod. Another point is, is systems init container validation enough to block api requests?

MStokluska avatar Jan 23 '25 14:01 MStokluska

There is a new functionality added to resync-routes when zync is enabled. This causes a job to be created, resync-route. The job keeps failing due to:

Failed to load CA Certificate or CA Path
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:37:in `initialize'
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:24:in `new'
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:24:in `ssl_context'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client/config.rb:125:in `ssl_context'
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:134:in `connect'
/opt/system/vendor/bundle/ruby/3.1.0/gems/hiredis-client-0.22.2/lib/redis_client/hiredis_connection.rb:50:in `initialize'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:746:in `new'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:746:in `block in connect'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client/middlewares.rb:12:in `connect'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:745:in `connect'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:732:in `raw_connection'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:697:in `ensure_connected'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-client-0.22.2/lib/redis_client.rb:292:in `call_v'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis/client.rb:90:in `call_v'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis.rb:152:in `block in send_command'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis.rb:151:in `synchronize'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis.rb:151:in `send_command'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis/commands/connection.rb:21:in `ping'
/opt/system/lib/tasks/boot.rake:26:in `block (3 levels) in <top (required)>'
/opt/system/vendor/bundle/ruby/3.1.0/gems/redis-5.3.0/lib/redis.rb:95:in `with'
/opt/system/app/lib/system/redis_pool.rb:26:in `public_send'
/opt/system/app/lib/system/redis_pool.rb:26:in `block in method_missing'

@MStokluska are you suggest to open a Jir

When enabling TLS with wrong certs (I've tried with storage) what happens is that although the authentication in backend listener fails, the products can be created and promoted to stage/prod. Auth fails on api requests. Then, when fixing up the certs, the requests are still failing. I believe this requires a redis re-sync? I guess, porta does some validation right, so maybe an idea here would be to restart system-app pod on every update of the TLS secrets to perform the validation (and hopefully catch the incorrect certs and block api access?) WDYT?

@MStokluska , all backend pods (listener, worker, cron) and system (app and sidekiq) are restarting if any change in secrets system-redis and backend-redis (that hold certificates), as whatched-by is available on backend-redis and system-redis

In your current implementation, watched-by doesn't work correctly. When I've fixed it, watched-by on backend-redis doesn't restart system pod. Another point is, is systems init container validation enough to block api requests?

@MStokluska , Current PR - Completed, tested, and confirmed to work exactly as specified in Jira/Requirements, as well as in alignment with our team prior meeting discussions and my prior conversation with System/Juan (see team chat for details). Regarding your suggestion to add additional Redis TLS validation to the Operator, this represents a change in the requirements. To address it, I've opened a follow-up task: THREESCALE-11453. Thank you for your suggestion. cc: @briangallagher .

valerymo avatar Jan 23 '25 14:01 valerymo

Verification

Test cases

  1. Initial install without TLS - all works
  2. Adding values to the relevant secret - nothing should happen and 3scale should still work as expected without tls
  3. Adding the watch-by to secret - operator should trigger but nothing should change because TLS flag on APIM is missing
  4. Removing one of the keys from secret and enabling TLS on APIM - Operator should not apply the envs but report in errors that crucial key is missing
  5. Re-adding the key back in, Operator should enable TLS on the deployments
  6. Confirming that it all works with TLS
  7. Disabling TLS on the APIM by setting flag to false - expectation is that the envs for TLS should disappear from the deployments and all should work without tls (unless you are forcing SSL on the dbs)

1. Initial install without TLS - all works

  • make cluster/prepare/local to install non tls db and redis
  • create s3 secret
  • create apimanager cr with external redis and system db
  • make run
  • install completes successfully
  • no additional env var tls added to backend or system or zync
  • no volume mounts created for tls
  • operator logs reporting one issue during startup
2025-02-27T10:49:55Z    INFO    olm     Found deployments with status   {"stopped": [], "starting": ["apicast-production", "system-app", "system-sidekiq", "zync", "zync-que"], "ready": ["apicast-staging", "backend-cron", "backend-listener", "backend-worker", "system-memcache", "system-searchd", "zync-database"]}
2025-02-27T10:49:55Z    DEBUG   controllers.APIManager  Status  {"Status Reconciler": {"name":"apimanager-sample","namespace":"3scale-test"}, "status is different": false}
2025-02-27T10:49:55Z    ERROR   Reconciler error        {"controller": "apimanager", "controllerGroup": "apps.3scale.net", "controllerKind": "APIManager", "APIManager": {"name":"apimanager-sample","namespace":"3scale-test"}, "namespace": "3scale-test", "name": "apimanager-sample", "reconcileID": "6e87da06-d638-4e54-97a0-d176d1125e44", "error": "Operation cannot be fulfilled on jobs.batch \"system-app-pre\": StorageError: invalid object, Code: 4, Key: /kubernetes.io/jobs/3scale-test/system-app-pre, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 9d500022-11f6-402f-a3c3-5f284d17fe44, UID in object meta: "}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /home/austincunningham/repo/3scale-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /home/austincunningham/repo/3scale-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /home/austincunningham/repo/3scale-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2025-02-27T10:50:09Z    INFO    controllers.APIManager  ReconcileAPIManager     {"apimanager": {"name":"apimanager-sample","namespace":"3scale-test"}, "Operator version": "0.13.0", "3scale release": "2.16"}

I don't believe it's an issue system-app-pre job was successful.

2. Adding values to the relevant secret (Nothing should happen)

adding values to backend-redis no change in the reconciler adding values to system-redis no change in the reconciler

3.Adding the watch-by to secret - operator should trigger but nothing should change because TLS flag on APIM is missing

adding label to secrets triggers a reconcile but nothing is added to the deployments

W0227 14:55:30.061454  220283 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
W0227 15:03:50.160624  220283 warnings.go:70] apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2025-02-27T15:05:06Z    DEBUG   controllers.APIManager.secretToApimanagerEventMapper    Processing object       {"key": {"name":"system-redis","namespace":"3scale-test"}, "accepted": false}
2025-02-27T15:05:06Z    DEBUG   controllers.APIManager.secretToApimanagerEventMapper    Processing object       {"key": {"name":"system-redis","namespace":"3scale-test"}, "accepted": false}
2025-02-27T15:06:10Z    DEBUG   controllers.APIManager.secretToApimanagerEventMapper    Processing object       {"key": {"name":"backend-redis","namespace":"3scale-test"}, "accepted": false}
2025-02-27T15:06:10Z    DEBUG   controllers.APIManager.secretToApimanagerEventMapper    Processing object       {"key": {"name":"backend-redis","namespace":"3scale-test"}, "accepted": false}

4.Removing one of the keys from secret and enabling TLS on APIM - Operator should not apply the envs but report in errors that crucial key is missing

removed the two CA certs from backend-redis and the CA cert from system redis We have blocked env var creation and volume mounts Minor issue with logging only logging backend-redis for a single env var even though 2 were removed. As this is blocking logic when the backend reports an issue we never see logs for system if both have issues. This is to be expected. When backend is corrected we see system errors.

2025-02-27T15:12:56Z    ERROR   Reconciler error        {"controller": "apimanager", "controllerGroup": "apps.3scale.net", "controllerKind": "APIManager", "APIManager": {"name":"apimanager-sample","namespace":"3scale-test"}, "namespace": "3scale-test", "name": "apimanager-sample", "reconcileID": "84ca2ff9-2dc4-43a3-9fcf-d026ced854b1", "error": "validation errors for Redis TLS configuration in 'backend-redis' secret: 'backendRedisTLSEnabled: true' is set in apimanager. Secret validation errors: [Secret field 'REDIS_SSL_CA' is required in secret 'backend-redis']"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /home/austincunningham/repo/3scale-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /home/austincunningham/repo/3scale-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /home/austincunningham/repo/3scale-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227

5. Re-adding the key back in, Operator should enable TLS on the deployments

  • created redis server with certs
  • created client side secrets for system-redis and backend-redis
  • ran the 3scale install
  • had to bypass the preflights
  • issue with the system-pre-app
rake aborted!
Redis::CannotConnectError: Resource temporarily unavailable (rediss://172.30.116.196:6380/2)

and sidekiq connecting to the database

rake aborted!
Redis::CannotConnectError: Resource temporarily unavailable (rediss://172.30.116.196:6380/2)

Will look again on Monday. After a good bit of debugging Looks like an compatibility issue with the openssl commands on my system compared to yours. openssl version OpenSSL 3.4.0 22 Oct 2024 (Library: OpenSSL 3.4.0 22 Oct 2024 mine openssl version OpenSSL 3.1.1 30 May 2023 (Library: OpenSSL 3.1.1 30 May 2023)

updated the openssl and looks like the certs are working now and the install completes

6. Confirming that it all works with TLS

all working with tls

7. Disabling TLS on the APIM by setting flag to false - expectation is that the envs for TLS should disappear from the deployments and all should work without tls (unless you are forcing SSL on the dbs)

This step fails , for system, setting the flag to false causes system-pre-app to fail along with sidekiq and searchd. the operator fails to finish installing components( could be due to the redis setup to enforce tls but not sure)

Same with disabling backend redis and queues , the backend fails to recover.

If its a misconfiguration in the secret regarding the connection url when the flag is set, the operator should pick this up and have a error in the logs to inform the user. Don't see any such message. fyi @valerymo

austincunningham avatar Feb 27 '25 15:02 austincunningham

@austincunningham , I rechecked case when 2 fields are missing in backend-redis secret. It's working as expected for me - both missing mandatory fields (**_CA) - reported as missing:


2025-02-27T19:03:26+02:00	ERROR	Reconciler error	{"controller": "apimanager", "controllerGroup": "apps.3scale.net", "controllerKind": "APIManager", "APIManager": {"name":"3scale","namespace":"3scale-test"}, "namespace": "3scale-test", "name": "3scale", "reconcileID": "ef627840-10e3-433e-aef0-b7a6e231a763", "error": "validation errors for Redis TLS configuration in 'backend-redis' secret: 'backendRedisTLSEnabled: true' is set in apimanager. Secret validation errors: [Secret field 'REDIS_SSL_CA' is required in secret 'backend-redis']\n'queuesRedisTLSEnabled: true' is set in apimanager. Secret validation errors: [Secret field 'REDIS_SSL_QUEUES_CA' is required in secret 'backend-redis']"}
  • secret fields changed for test, "_1" suffix was added to simulate missied fields.
 oc describe secret backend-redis |grep "CA"
REDIS_SSL_QUEUES_CA_1:         1123 bytes
REDIS_SSL_CA_1:                1123 bytes

valerymo avatar Feb 27 '25 17:02 valerymo

hey @austincunningham , I tested the switch from NON-TLS to TLS, and it worked fine on my end. Could you please review the logs and the steps in the order outlined below. Thank you!

test_03Mar.txt

NO TLS

  • RedisTLSEnabled: false
DOMAIN=$(oc get routes console -n openshift-console -o json | jq -r '.status.ingress[0].routerCanonicalHostname' | sed 's/router-default.//')

cat << EOF | oc create -f -
apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
  name: 3scale
spec:
  system: 
    systemRedisTLSEnabled: false
    fileStorage: 
      simpleStorageService: 
        configurationSecretRef: 
          name: s3-credentials
  backend:
    backendRedisTLSEnabled: false
    queuesRedisTLSEnabled: false    
  wildcardDomain: $DOMAIN
  externalComponents:
    backend:
      redis: true
    system:
      redis: true
      database: true
EOF
  • Secrets URLs are not secure: redis, port: 6379
oc get secret system-redis -oyaml |grep URL |awk '{print $2}' |base64 -d
redis://172.30.60.33:6379/2

oc get secret backend-redis -oyaml |grep URL| awk '{print $2}' |base64 -d
redis://172.30.60.33:6379/1 redis://172.30.60.33:6379/0  

oc get secret backend-redis -oyaml | grep watched
    apimanager.apps.3scale.net/watched-by: apimanager

oc get secret system-redis -oyaml | grep watched
    apimanager.apps.3scale.net/watched-by: apimanager
  • All is UP:
oc get deploy                               
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
apicast-production   1/1     1            1           3m57s
apicast-staging      1/1     1            1           3m58s
backend-cron         1/1     1            1           5m12s
backend-listener     1/1     1            1           5m12s
backend-worker       1/1     1            1           5m11s
redis                1/1     1            1           50m
system-app           1/1     1            1           3m26s
system-memcache      1/1     1            1           5m10s
system-postgresql    1/1     1            1           52m
system-searchd       1/1     1            1           5m9s
system-sidekiq       1/1     1            1           3m59s
zync                 1/1     1            1           3m58s
zync-database        1/1     1            1           3m58s
zync-que             1/1     1            1           3m58s

oc get pod
NAME                                     READY   STATUS      RESTARTS        AGE
apicast-production-688bcc4cb6-cw4pd      1/1     Running     0               4m32s
apicast-staging-766b467df6-dxs2q         1/1     Running     0               4m33s
backend-cron-84b544d6d5-4ml55            1/1     Running     0               5m47s
backend-listener-c5bc44b86-652h7         1/1     Running     0               5m44s
backend-worker-7b8c4f9875-lh9jm          1/1     Running     0               5m44s
redis-8597cf58f5-jjtkj                   1/1     Running     0               29m
system-app-766fb58585-ndtcl              3/3     Running     0               3m59s
system-app-post-sjx6k                    0/1     Completed   0               2m58s
system-app-pre-529lj                     0/1     Completed   0               3m55s
system-memcache-7549c94bb7-4dpmg         1/1     Running     0               5m45s
system-postgresql-7d6b75f659-k4r58       1/1     Running     0               52m
system-searchd-55667cc76b-s9skx          1/1     Running     0               5m44s
system-searchd-manticore-reindex-t8skh   0/1     Completed   0               4m34s
system-sidekiq-86546688fd-lpfm8          1/1     Running     0               4m31s
zync-5654c58bf6-rxlk6                    1/1     Running     0               4m33s
zync-database-979844448-mwc2f            1/1     Running     0               4m33s
zync-que-5d5d5df44f-qrr29                1/1     Running     2 (4m22s ago)   4m33s

date
Mon Mar  3 13:01:16 EET 2025

switch to TLS

  1. Change secrets URLs from redis://172.30.60.33:6379/x to rediss://172.30.60.33:6380/x Note. After secrets URLs updated - nothing changed.
oc get secret system-redis -oyaml |grep URL |awk '{print $2}' |base64 -d
rediss://172.30.60.33:6380/2

oc get secret backend-redis -oyaml |grep URL |awk '{print $2}' |base64 -d 
rediss://172.30.60.33:6380/1 rediss://172.30.60.33:6380/0
  1. change APIManager CR, set: backendRedisTLSEnabled: true queuesRedisTLSEnabled: true systemRedisTLSEnabled: true
  • All is UP, update done
oc get pod                                       
NAME                                     READY   STATUS      RESTARTS      AGE
apicast-production-688bcc4cb6-cw4pd      1/1     Running     0             20m
apicast-staging-766b467df6-dxs2q         1/1     Running     0             20m
backend-cron-d54d98786-kmchz             1/1     Running     0             2m6s
backend-listener-748f64944-lvx75         1/1     Running     0             2m6s
backend-worker-78d9484df6-4vbmk          1/1     Running     0             2m6s
redis-8597cf58f5-jjtkj                   1/1     Running     0             45m
system-app-f5dd55bfd-r5qsx               3/3     Running     0             2m5s
system-app-post-c6bq6                    0/1     Completed   0             64s
system-app-pre-xfnnx                     0/1     Completed   0             2m4s
system-memcache-7549c94bb7-4dpmg         1/1     Running     0             21m
system-postgresql-7d6b75f659-k4r58       1/1     Running     0             68m
system-searchd-55667cc76b-s9skx          1/1     Running     0             21m
system-searchd-manticore-reindex-t8skh   0/1     Completed   0             20m
system-sidekiq-8d7b8c855-2fpcq           1/1     Running     0             2m5s
zync-5654c58bf6-rxlk6                    1/1     Running     0             20m
zync-database-979844448-mwc2f            1/1     Running     0             20m
zync-que-5d5d5df44f-qrr29                1/1     Running     2 (20m ago)   20m
check env vars in backend and system
  1. system-sidekiq
oc rsh system-sidekiq-8d7b8c855-2fpcq
Defaulted container "system-sidekiq" out of: system-sidekiq, check-svc (init)
sh-5.1$ env |grep -E "REDIS_CLIENT_CERT|BACKEND_REDIS_CLIENT_CERT"
env |grep -E "REDIS_CA_FILE|BACKEND_REDIS_CA_FILE"
env |grep -E "REDIS_PRIVATE_KEY|BACKEND_REDIS_PRIVATE_KEY"
env |grep -E "REDIS_SSL|BACKEND_REDIS_SSL"
REDIS_CLIENT_CERT=/tls/system-redis/system-redis-client.crt
BACKEND_REDIS_CLIENT_CERT=/tls/backend-redis/backend-redis-client.crt
REDIS_CA_FILE=/tls/system-redis/system-redis-ca.crt
BACKEND_REDIS_CA_FILE=/tls/backend-redis/backend-redis-ca.crt
REDIS_PRIVATE_KEY=/tls/system-redis/system-redis-private.key
BACKEND_REDIS_PRIVATE_KEY=/tls/backend-redis/backend-redis-private.key
REDIS_SSL=1
BACKEND_REDIS_SSL=1
sh-5.1$ cat /tls/backend-redis/backend-redis-ca.crt
-----BEGIN CERTIFICATE-----
MIIDDzCC......

 oc logs  system-sidekiq-8d7b8c855-2fpcq |less

        libjemalloc.so.2 (libc6,x86-64) => /usr/local/lib64/libjemalloc.so.2
I, [2025-03-03T11:15:33.560187 #2]  INFO -- : ActiveMerchant MODE set to 'production'
I, [2025-03-03T11:15:33.832569 #2]  INFO -- : [Core] Using http://backend-listener:3000/internal/ as URL
W, [2025-03-03T11:15:34.655860 #2]  WARN -- : OpenIdAuthentication.store is nil. Using in-memory store.
W, [2025-03-03T11:15:34.658288 #2]  WARN -- [Bugsnag]: No valid API key has been set, notifications will not be sent
2025-03-03T11:15:34.768Z pid=2 tid=66u INFO: Sidekiq 7.3.2 connecting to Redis with options {:size=>10, :pool_name=>"internal", :db=>"2", :ssl=>true, :url=>"rediss://172.30.60.33:6380/2"}
  1. system-app
 oc rsh  system-app-f5dd55bfd-r5qsx  
Defaulted container "system-master" out of: system-master, system-provider, system-developer
sh-5.1$ env |grep -E "REDIS_CLIENT_CERT|BACKEND_REDIS_CLIENT_CERT"
env |grep -E "REDIS_CA_FILE|BACKEND_REDIS_CA_FILE"
env |grep -E "REDIS_PRIVATE_KEY|BACKEND_REDIS_PRIVATE_KEY"
env |grep -E "REDIS_SSL|BACKEND_REDIS_SSL"
REDIS_CLIENT_CERT=/tls/system-redis/system-redis-client.crt
BACKEND_REDIS_CLIENT_CERT=/tls/backend-redis/backend-redis-client.crt
REDIS_CA_FILE=/tls/system-redis/system-redis-ca.crt
BACKEND_REDIS_CA_FILE=/tls/backend-redis/backend-redis-ca.crt
REDIS_PRIVATE_KEY=/tls/system-redis/system-redis-private.key
BACKEND_REDIS_PRIVATE_KEY=/tls/backend-redis/backend-redis-private.key
REDIS_SSL=1
BACKEND_REDIS_SSL=1
sh-5.1$
  1. system-app-pre
 oc logs system-app-pre-xfnnx
	libjemalloc.so.2 (libc6,x86-64) => /usr/local/lib64/libjemalloc.so.2
I, [2025-03-03T11:15:19.270217 #2]  INFO -- : ActiveMerchant MODE set to 'production'
....
Backend Internal API version 3.4.3 status: ok
Connected to postgresql://[email protected]/dev
Connected to rediss://172.30.60.33:6380/2
  1. backend - worker
 oc rsh backend-worker-78d9484df6-4vbmk
Defaulted container "backend-worker" out of: backend-worker, backend-redis-svc (init)
sh-4.4$ env |grep -E "CONFIG_REDIS_CA_FILE|CONFIG_QUEUES_CA_FILE"  
CONFIG_QUEUES_CA_FILE=/tls/queues/config-queues-ca.crt
CONFIG_REDIS_CA_FILE=/tls/backend-redis-ca.crt
sh-4.4$ env |grep -E "CONFIG_REDIS_CERT|CONFIG_QUEUES_CERT"
CONFIG_QUEUES_CERT=/tls/queues/config-queues-client.crt
CONFIG_REDIS_CERT=/tls/backend-redis-client.crt
sh-4.4$ env |grep -E "CONFIG_REDIS_PRIVATE_KEY|CONFIG_QUEUES_PRIVATE_KEY"
CONFIG_REDIS_PRIVATE_KEY=/tls/backend-redis-private.key
CONFIG_QUEUES_PRIVATE_KEY=/tls/queues/config-queues-private.key
sh-4.4$ env |grep -E "CONFIG_REDIS_SSL|CONFIG_QUEUES_SSL"
CONFIG_QUEUES_SSL=1
CONFIG_REDIS_SSL=1
sh-4.4$

valerymo avatar Mar 03 '25 11:03 valerymo

/test test-unit

valerymo avatar Mar 06 '25 13:03 valerymo

@austincunningham - I retested after small update: 1) TLS -> NonTLS-> TLS 2) NonTLS->TLS->NonTls. 3) redis URLs validation - errors reported, All is continue running and "switched" only after URLs updated correctly in secrets. All looks good for me Thank you

valerymo avatar Mar 10 '25 07:03 valerymo