concourse-chart icon indicating copy to clipboard operation
concourse-chart copied to clipboard

Helm Upgrade -- Error: UPGRADE FAILED: cannot patch "concourse-ci-postgresql" with kind StatefulSet

Open mlopez-eb opened this issue 3 years ago • 2 comments

Made a simple change to values.yaml file to enable enableAcrossStep. Ran the following upgrade command & received the error:

> helm upgrade concourse-ci concourse/concourse -f values.yaml --namespace=concourse
W0308 10:04:22.775295    6315 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0308 10:04:23.057213    6315 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0308 10:04:23.423974    6315 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
Error: UPGRADE FAILED: cannot patch "concourse-ci-postgresql" with kind StatefulSet: StatefulSet.apps "concourse-ci-postgresql" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden

> helm version
version.BuildInfo{Version:"v3.8.0", GitCommit:"d14138609b01886f544b2025f5000351c9eb092e", GitTreeState:"clean", GoVersion:"go1.17.6"}

> helm list
NAME        	NAMESPACE   	REVISION	UPDATED                             	STATUS  	CHART              	APP VERSION
concourse-ci	concourse   	12      	2022-03-08 10:04:14.032308 -0500 EST	failed  	concourse-16.0.3   	7.5.0

> fly -v
7.0.0

> kubectl version --short
Client Version: v1.23.4
Server Version: v1.21.5-eks-bc4871b

mlopez-eb avatar Mar 08 '22 15:03 mlopez-eb

+1, issue is still present with chart v16.1.24

philippeboyd avatar Jun 06 '22 18:06 philippeboyd

PostgreSQL chart dependency was updated with https://github.com/concourse/concourse-chart/commit/c2a35012d7df65869fe5ccf9f0ae35632134b98b.

This upgrade requires some manual ❤️ as it comes with two breaking changes [1]:

  1. the chart itself (therefore the Forbidden: updates to statefulset spec for fields other than 'replicas',... error)
  2. it bumps postgres from v11 to v14 (which requires data migration)

Migrate the chart

To apply the chart without requiring to migrate to postgres v14 pin the postgres image to the current version (eg. 11.14.0-debian-10-r28):

values.yaml:

concourse:
  postgresql:
    image:
      tag: 11.14.0-debian-10-r28
  1. 💾 backup (in case anything goes wrong)
  2. 🪣 delete the concourse-postgresql StatefulSet
  3. 🚀 then apply the Chart with the updates values.yaml

You're basically done here if you don't want to upgrade postgres itself. Otherwise continue with [2].

See also:

  • [1] https://docs.bitnami.com/kubernetes/infrastructure/postgresql/administration/upgrade/#to-1100
  • [2] https://www.postgresql.org/docs/current/upgrading.html

bookwalker avatar Jun 12 '22 09:06 bookwalker