Various charts depending on deprecated kubernetes APIs

[sochotnicky]

I was looking specifically at PodSecurityPolicy, but others might be impacting various charts too.

As per: https://kubernetes.io/docs/reference/using-api/deprecation-guide/#psp-v125

The policy/v1beta1 API will not be served from Kubernetes 1.25 onward. It's possible other deprecated endpoints are used too.

$ git grep 'policy/v1beta1' 
charts/enterprise-metrics/templates/pdb.yaml:apiVersion: policy/v1beta1
charts/enterprise-metrics/templates/podsecuritypolicy.yaml:apiVersion: policy/v1beta1
charts/fluent-bit/templates/podsecuritypolicy.yaml:apiVersion: policy/v1beta1
charts/grafana/templates/poddisruptionbudget.yaml:apiVersion: policy/v1beta1
charts/grafana/templates/podsecuritypolicy.yaml:apiVersion: policy/v1beta1
charts/grafana/templates/tests/test-podsecuritypolicy.yaml:apiVersion: policy/v1beta1
charts/loki-distributed/templates/distributor/poddisruptionbudget-distributor.yaml:apiVersion: policy/v1beta1
charts/loki-distributed/templates/gateway/poddisruptionbudget-gateway.yaml:apiVersion: policy/v1beta1
charts/loki-distributed/templates/index-gateway/poddisruptionbudget-index-gateway.yaml:apiVersion: policy/v1beta1
charts/loki-distributed/templates/ingester/poddisruptionbudget-ingester.yaml:apiVersion: policy/v1beta1
charts/loki-distributed/templates/memcached-chunks/poddisruptionbudget-memcached-chunks.yaml:apiVersion: policy/v1beta1
charts/loki-distributed/templates/memcached-frontend/poddisruptionbudget-memcached-frontend.yaml:apiVersion: policy/v1beta1
charts/loki-distributed/templates/memcached-index-queries/poddisruptionbudget-memcached-index-queries.yaml:apiVersion: policy/v1beta1
charts/loki-distributed/templates/memcached-index-writes/poddisruptionbudget-memcached-index-writes.yaml:apiVersion: policy/v1beta1
charts/loki-distributed/templates/podsecuritypolicy.yaml:apiVersion: policy/v1beta1
charts/loki-distributed/templates/querier/poddisruptionbudget-querier.yaml:apiVersion: policy/v1beta1
charts/loki-distributed/templates/query-frontend/poddisruptionbudget-query-frontent.yaml:apiVersion: policy/v1beta1
charts/loki-distributed/templates/ruler/poddisruptionbudget-ruler.yaml:apiVersion: policy/v1beta1
charts/loki/templates/pdb.yaml:apiVersion: policy/v1beta1
charts/loki/templates/podsecuritypolicy.yaml:apiVersion: policy/v1beta1
charts/promtail/templates/podsecuritypolicy.yaml:apiVersion: policy/v1beta1

This will prevent various components to be deployed in Kubernetes 1.25 or higher (using helm charts anyway)

[cyclelabs-ryanberger]

This also is currently causing charts warning messages that mark them as failures when using azure devops to deploy charts. I hope this can be resolved soon.

On Wed, Oct 20, 2021 at 9:22 AM Stanislav Ochotnický < @.***> wrote:

I was looking specifically at PodSecurityPolicy, but others might be impacting various charts too.

As per: https://kubernetes.io/docs/reference/using-api/deprecation-guide/#psp-v125

The policy/v1beta1 API will not be served from Kubernetes 1.25 onward. It's possible other deprecated endpoints are used too.

$ git grep 'policy/v1beta1' charts/enterprise-metrics/templates/pdb.yaml:apiVersion: policy/v1beta1 charts/enterprise-metrics/templates/podsecuritypolicy.yaml:apiVersion: policy/v1beta1 charts/fluent-bit/templates/podsecuritypolicy.yaml:apiVersion: policy/v1beta1 charts/grafana/templates/poddisruptionbudget.yaml:apiVersion: policy/v1beta1 charts/grafana/templates/podsecuritypolicy.yaml:apiVersion: policy/v1beta1 charts/grafana/templates/tests/test-podsecuritypolicy.yaml:apiVersion: policy/v1beta1 charts/loki-distributed/templates/distributor/poddisruptionbudget-distributor.yaml:apiVersion: policy/v1beta1 charts/loki-distributed/templates/gateway/poddisruptionbudget-gateway.yaml:apiVersion: policy/v1beta1 charts/loki-distributed/templates/index-gateway/poddisruptionbudget-index-gateway.yaml:apiVersion: policy/v1beta1 charts/loki-distributed/templates/ingester/poddisruptionbudget-ingester.yaml:apiVersion: policy/v1beta1 charts/loki-distributed/templates/memcached-chunks/poddisruptionbudget-memcached-chunks.yaml:apiVersion: policy/v1beta1 charts/loki-distributed/templates/memcached-frontend/poddisruptionbudget-memcached-frontend.yaml:apiVersion: policy/v1beta1 charts/loki-distributed/templates/memcached-index-queries/poddisruptionbudget-memcached-index-queries.yaml:apiVersion: policy/v1beta1 charts/loki-distributed/templates/memcached-index-writes/poddisruptionbudget-memcached-index-writes.yaml:apiVersion: policy/v1beta1 charts/loki-distributed/templates/podsecuritypolicy.yaml:apiVersion: policy/v1beta1 charts/loki-distributed/templates/querier/poddisruptionbudget-querier.yaml:apiVersion: policy/v1beta1 charts/loki-distributed/templates/query-frontend/poddisruptionbudget-query-frontent.yaml:apiVersion: policy/v1beta1 charts/loki-distributed/templates/ruler/poddisruptionbudget-ruler.yaml:apiVersion: policy/v1beta1 charts/loki/templates/pdb.yaml:apiVersion: policy/v1beta1 charts/loki/templates/podsecuritypolicy.yaml:apiVersion: policy/v1beta1 charts/promtail/templates/podsecuritypolicy.yaml:apiVersion: policy/v1beta1

This will prevent various components to be deployed in Kubernetes 1.25 or higher (using helm charts anyway)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/grafana/helm-charts/issues/748, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOLST6X34WANMDWRCWTBCNLUH27CRANCNFSM5GLUNSQQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[sboschman]

Just to start with the Grafana chart:

PodDisruptionBudget is a simple bump to policy/v1 from 1.21+ (selector is explicitly defined in the charts, so no issue with the empty selector change).

PodSecurityPolicy is completely removed... It is already possible to disable it. Should the grafana chart be changed to explicitly disable it when running on 1.25+? Not sure what everyone expects to happen? Maybe when 1.25 goes live the default pspEnabled=true can be changed to pspEnabled=false, so people on older k8s versions can enabled it again if they want?

[torstenwalter]

All valid input. I propose the following approach to resolve it:

1. add a second build pipeline which tests various kubernetes versions. I would go for a second pipeline as it would fail for deprecated APIs until all issues are fixed. By having a seperate pipeline for that we would get CI feedback, but it would be non blocking. So reviewer can decide to merge without it. Otherwise one would force the next person who fixes a typo also to resolve all deprecations etc. Note: Pipeline from Jenkins chart could serve as an example https://github.com/jenkinsci/helm-charts/blob/main/.github/workflows/lint-test.yaml#L10-L18

2. create individual PRs for charts which fix deprecations. Ideally those PRs link to this issue and are small so easy to review.

3. keep track in this issue which charts are already completely fixed - pipeline which checks different k8s versions succeeds. We could use a simple table with the name of the charts for that purpose.

4. once all charts are fixed we make replace the old build pipeline with the new once pipeline which checks the different k8s versions. That way it's mandatory to pass it.

[sboschman]

To slightly complicate this issue, I ran into a Helm issue regarding the detection of deprecations.

[sboschman]

Another complication is the fact that chart-testing does not support --strict (issue). Deprecations reported by Helm lint are warnings, not errors. The --strict also fails Helm lint on warnings and allows failing the ci build when deprecations are found.

[cyclelabs-ryanberger]

Hi all,

Is there any update on a resolution for this going forward?

Just to start with the Grafana chart:

PodDisruptionBudget is a simple bump to policy/v1 from 1.21+ (selector is explicitly defined in the charts, so no issue with the empty selector change).

PodSecurityPolicy is completely removed... It is already possible to disable it. Should the grafana chart be changed to explicitly disable it when running on 1.25+? Not sure what everyone expects to happen? Maybe when 1.25 goes live the default pspEnabled=true can be changed to pspEnabled=false, so people on older k8s versions can enabled it again if they want?

Can this just be changed to false to omit this error message?

[sboschman]

Can this just be changed to false to omit this error message?

This blog post explains the PodSecurityPolicy Deprecation in more detail.

[ATarrio]

End of life for v1.21 on GKE is Dec 2022, so users of GKE must upgrade to at least 1.22 before that. https://cloud.google.com/kubernetes-engine/docs/release-schedule

[guidoiaquinti]

This is likely going to become a bigger issue since k8s 1.25 has been announced yesterday Announcing the release of Kubernetes v1.25

[bjornbouetsmith]

Any update on this - the grafana chart and others that use this will stop working if anyone upgrade to 1.25 of kubernetes.

[tamanjoku]

No updates yet on this? I want to upgrade to 1.25, but very weary about this.

[sboschman]

The Grafana chart was fixed in #1500 , wasn't it?

[tamanjoku]

Still getting the message when I run the helm charts.

[bjornbouetsmith]

The Grafana chart was fixed in #1500 , wasn't it?

What version of the helm chart does that translate to? i.e. what version and above?

Still getting the message when I run the helm charts.

I have updated my chart with: rbac: pspEnabled: false

And then I just deleted the PodSecurity policies and everything works just fine.

But I agreee that it would be awesome to have an updated helm chart without this issue.

[Danpiel]

Hello, PodSecurityPolicy issue comes in k8s 1.26 with grafana, even with rbac.pspEnabled=false option. Chart version 6.50 and onwards are resulting this error

[cvallesi-kainos]

We are trying to update our clusters to 1.25.6 but grafana (as well as loki) chart is preventing the update as per the PodSecurityPolicy even with pspEnabled = false.

An update to the chart is greatly needed.

Edit: adding that deploying from scratch is instead working as the policies are not created