OpenShift: upgrade action is not doing anything

[je-munobia]

Describe the bug I am attempting to perform an "upgrade" action with upgradeType set to "in-place". I expected a Job to start with a POD that executes the upgrade. The actual effect is that no job/pod is created. I was trying to find OpenShift examples, but the docs do not cover OpenShift.

To Reproduce

1. Install 8.5.2 (works)
2. Deploy 8.5.2 (works)
3. Upgrade 8.5.3 (no output)

Expected behavior I expected a Job to start with a POD that executes the upgrade. The actual effect is that no job/pod is created.

Chart version pega-1.5.3 No customizations to chart, only setting values.

Server (if applicable, please complete the following information):

• OS: RHEL 7.9 (Where I run helm 3.5.0)
• Environment: OpenShift 4.7.7
• Database: PostgreSQL 11

Additional context Command: helm upgrade coe-bat pega/pega -f charts/pega/values-postgresql-generic.yaml -f charts/pega/values-openshift-local-postgresql-coe-bat.yaml --set global.actions.execute=upgrade --set global.jdbc.username=pega --set global.jdbc.password=SomePwd --timeout 30m0s

values-postgresql-generic:

---
global:
  provider: "openshift"

  jdbc:
    driverClass: "org.postgresql.Driver"
    dbType: "postgres"
    # Baked into a bespoke image
    driverUri: "file:///opt/pega/xyz/postgresql-42.2.20.jar"

    rulesSchema: "rules"
    dataSchema: "data"

  docker:
    registry:
      url: "pega-docker-cicd-local.repository.xyz.net"
      username: ""
      password: ""
    pega:
      imagePullPolicy: IfNotPresent

#  upgrade:
#    kube-apiserver:
#      serviceHost: ""
#      httpsServicePort: ""

pegasearch:
  # Not allowed in OpenShift - Invalid value: true: Privileged containers are not allowed
  # I assume that the image comes with a properly configured /etc/sysctl.conf, vm.max_map_count=262144
  # TODO: Check with Pega if and why search image would not come correctly configured and what I should do.
  set_vm_max_map_count: false

cassandra:
  enabled: false

hazelcast:
  enabled: false

values-openshift-local-postgresql-coe-bat:

---
global:
  deployment:
    name: coe-bat

  jdbc:
    url: "jdbc:postgresql://my-postgresql-postgresql-0.my-postgresql-headless:5432/pega"

  docker:
    pega:
      image: "pega-docker-cicd-local.repository.xyz.net/platform/postgresql_pega:8.5.3-210521-1"

  tier:
    - name: "all"
      nodeType: "Stream,BackgroundProcessing,WebUser,Search"

      service:
        port: 80
        targetPort: 8080
        serviceType: LoadBalancer

      replicas: 2
      pegaDiagnosticUser: "pega_admin"
      pegaDiagnosticPassword: "SomePwd"
      volumeClaimTemplate:
        resources:
          requests:
            storage: 5Gi

pegasearch:
  image: "pega-docker-cicd-local.repository.xyz.net/platform/search:8.5.3"

installer:
  image: "pega-docker-cicd-local.repository.xyz.net/platform/postgresql_installer:8.5.3-210521-1"
  adminPassword: "SomePwd"
  upgrade:
    upgradeType: "in-place"

[je-munobia]

Since I get no feedback. I took a further looks and it seems the upgrade job is hooked up to the helm post-install phase. This got me suspicious, maybe the job needs also post-upgrade.

annotations:
  "helm.sh/hook": post-install,post-upgrade

The normal way for me to proceed would be:

1. helm install ... --set global.actions.execute=install ...
2. helm upgrade ... --set global.actions.execute=upgrade ...

When I do, the hook does not seem to trigger.

Is there some special reason why the hook should only trigger on helm install?

[je-munobia]

Perhaps fix here: charts/pega/charts/installer/templates/_pega-installer-job.tpl#L11

[je-munobia]

Possibly caused by this commit: https://github.com/pegasystems/pega-helm-charts/commit/e4441b7a347f23c4942b2ed36807642a7a2793e3

[MadhuriArugula]

@je-munobia You caught that right. It is an issue with the recent changes in the installer chart. Thanks for reporting the issue. It wasn't intentional to have only post-install hook. We will fix it as a bug.

[je-munobia]

Also: Default value here: charts/pega/charts/installer/values.yaml#L34 will become a string of legth 5 that evaluated to true if I am not mistaken... If that is the case then it is a bit misleading...

[je-munobia]

There is a workaround for those who have to work with the chart until the bug is fixed. In one of your values files or as a --set option to helm make sure that installer.waitForJobCompletion is set to false rather than "false". This will ensure that the erroneous waiting code is bypassed.

[MadhuriArugula]

@je-munobia We have fixed the issue with waitForJobCompletion in recent PR's and the change is already in master. You should be receiving the fix in newer helm release which will be out in a day or two. I'll keep this thread posted soon after the release is created.

[je-munobia]

@MadhuriArugula Great - that was fast! I took a look at master and noted that https://github.com/pegasystems/pega-helm-charts/blob/master/charts/pega/charts/installer/templates/_pega-installer-job.tpl#L11 might need to be changed to: "helm.sh/hook": post-install,post-upgrade

[MadhuriArugula]

@MadhuriArugula Great - that was fast! I took a look at master and noted that https://github.com/pegasystems/pega-helm-charts/blob/master/charts/pega/charts/installer/templates/_pega-installer-job.tpl#L11 might need to be changed to: "helm.sh/hook": post-install,post-upgrade @je-munobia We already have this in our work queue. The changes should be there in master after we perform a regression testing. Until then as waitForJobCompletion changes are already in master, it should unblock the clients.

[MadhuriArugula]

The issue has been addressed as part of #312 available from helm chart version v1.6.2