Helm charts repository index.yaml retention policy

[carrodher]

As reported in this issue (https://github.com/bitnami/charts/issues/8433), in the last few times we are facing some issues with the index.yaml associated with the Bitnami Helm charts repository.

Current situation

After some investigation, it seems the root cause is related to CloudFront reaching some limits due to the volume of traffic when serving the index.yaml. This index.yaml contains all the Bitnami Helm charts history (around 15300 entries), producing a pretty fat 14MB file. Given the size of the file and the volume of traffic, thousands of terabytes of download traffic per month are being generated.

One of the alternatives considered was the use of compression at CloudFront, in that case, this solution doesn’t work since compression is not used by the Helm client (helm) itself (see https://github.com/helm/helm/pull/8070) so it doesn’t solve the reported issue.

Mitigation

As the first line of action, we will reduce the size of the index.yaml by removing some old versions and keeping all versions for a period of time (6 months).

⚠️ Please note this action is not removing/deleting any Helm chart, packaged tarballs (.tgz) won't be removed, this action is only affecting index.yaml used to list the Helm charts. Previous versions of the index.yaml can be used to install old Helm charts.

Please note Helm charts tarballs (.tgz) won't be removed, this action is only affecting index.yaml.

Result

Applying this approach (https://github.com/bitnami/charts/pull/10530), we obtained the following results:

Total chart versions
* Before: 15260
* Removed: 12138
* After: 3122

Producing a reduced 3.5MB index.yaml.

🔧 Workaround for previous versions

The index.yaml is stored in this repository under the index branch, users should be able to use any commit in that branch to add a previous version of the index.yaml.

• Manually using helm repo add

$ helm repo add bitnami-full-index https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
"bitnami-full-index" has been added to your repositories

• When used as a dependency in Chart.yaml:

  - name: postgresql
    version: 8.1.0
-   repository: https://charts.bitnami.com/bitnami
+   repository: https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
    condition: postgresql.enabled

[DmytroDrachov]

Thank you very much

[sisrael-dn]

Hi, https://charts.bitnami.com/bitnami seems to return 403 AccessDenied error at the moment. Is this related to this issue? My guess is that if you work with S3 bucket as CloudFront origin, the new index.yaml got the wrong ACL now (not publicly readable)

[potiuk]

This is TERRIBLY wrong. For example, we released Helm Chart to our users https://airflow.apache.org/docs/helm-chart/stable/index.html with bitnami's Postgres 10.5.3 and all the charts our users are using are useless now if they use embedded Postgres.

How is it possible you do such a breaking change in such a way ? This must be some kind of a joke.

This totally undermines any trust in bitnami as a source of charts - You basically rendered all historical charts released by likely huge number of projects useless.

[bschurig]

As reported in this issue (#8433), in the last few times we are facing some issues with the index.yaml associated with the Bitnami Helm charts repository.

Current situation

After some investigation, it seems the root cause is related to CloudFront reaching some limits due to the volume of traffic when serving the index.yaml. This index.yaml contains all the Bitnami Helm charts history (around 15300 entries), producing a pretty fat 14MB file. Given the size of the file and the volume of traffic, thousands of terabytes of download traffic per month are being generated.

Mitigation

As the first line of action, we will reduce the size of the index.yaml by removing some old versions and keeping all versions for a period of time (6 months).

Please note Helm charts tarballs (.tgz) won't be removed, this action is only affecting index.yaml.

Result

Applying this approach (#10530), we obtained the following results:

Total chart versions
* Before: 15260
* Removed: 12138
* After: 3122

Producing a reduced 3.5MB index.yaml.

Workaround for previous versions

The index.yaml is stored in this repository under the index branch, users should be able to use any commit in that branch to add a previous version of the index.yaml, for example:

$ helm repo add bitnami-pre-2022 https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
"bitnami-pre-2022" has been added to your repositories

It would be fantastic if someone could get this added to the readme.

[GMZwinge]

Instead of removing all versions older than 6 months, would it be possible to keep the last major/minor versions older than 6 months, eg: mongodb-9.2.6.

[beltran-rubo]

Hi @potiuk, we apologise for the inconvenience. The Helm charts you built & distributed previously still are working properly for users, the previous version of the Helm chart is included into the tarball itself.

The part that is affected is the helm package or helm dependency update actions as part of the packaging process. All the charts are publicly available, all versions. The workaround mentioned into the post allows you to continue building the Helm chart with the exact same version.

[adawalli]

Hi @potiuk, we apologise for the inconvenience. The Helm charts you built & distributed previously still are working properly for users, the previous version of the Helm chart is included into the tarball itself.

The part that is affected is the helm package or helm dependency update actions as part of the packaging process. All the charts are publicly available, all versions. The workaround mentioned into the post allows you to continue building the Helm chart with the exact same version.

Why didn't you consider a proper CDN or basically...anything else? You make it seem like this is no big deal and that there is an existing easy path for people. The reality is that a ton of pipelines, dev flows, and deployments have just broken and people are scrambling to find out why. There is no way to spin that this was a good decision as executed - this was done extremely rashly.

[CrisFavero]

The Url in this code snippet for your fix returns 404

$ helm repo add bitnami-pre-2022 https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
"bitnami-pre-2022" has been added to your repositories

[nabilbendafi]

$ helm repo add bitnami-pre-2022 https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami "bitnami-pre-2022" has been added to your repositories

@CrisFavero Normal. Because helm will look for an index.yaml in this base url

[adawalli]

The Url in this code snippet for your fix returns 404

$ helm repo add bitnami-pre-2022 https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
"bitnami-pre-2022" has been added to your repositories

redacting incorrect info - the above info is correct

[nabilbendafi]

@CrisFavero

A chart repository is an HTTP server that houses an index.yaml file and optionally some packaged charts. When you're ready to share your charts, the preferred way to do so is by uploading them to a chart repository.

So the current workaround simply uses githubusercontent.com as a free webserver to provide that essential/critical index.yaml, that contains the links to the helm charts versions we all need.

[Pokom]

@adawalli if you don't specify the commit hash then you're using the latest index, which does not include older releases. @CrisFavero this URI worked for me: https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami

[adawalli]

@adawalli if you don't specify the commit hash then you're using the latest index, which does not include older releases. @CrisFavero this URI worked for me: https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami

Thanks - you are correct - I redacted my previous comment to prevent confusion

[carrodher]

In order to list old versions you need to use a previous commit, see below the differences:

$ helm repo list
NAME            	URL
bitnami         	https://charts.bitnami.com/bitnami
bitnami-index   	https://raw.githubusercontent.com/bitnami/charts/index/bitnami
bitnami-pre-2022	https://raw.githubusercontent.com/bitnami/charts/pre-2022/bitnami

• bitnami (latest 6 months):

$ helm search repo bitnami/common -l
NAME          	CHART VERSION	APP VERSION	DESCRIPTION
bitnami/common	1.15.1       	1.15.1     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.14.2       	1.14.2     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.14.1       	1.13.1     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.14.0       	1.13.1     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.13.1       	1.13.1     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.13.0       	1.13.0     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.12.0       	1.12.0     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.11.3       	1.11.3     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.11.2       	1.11.2     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.11.1       	1.11.1     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.11.0       	1.11.0     	A Library Helm Chart for grouping common logic ...
bitnami/common	1.10.4       	1.10.0     	A Library Helm Chart for grouping common logic ...

• bitnami-index, same as bitnami (latest 6 months), but using GitHub to serve it:

$ helm search repo bitnami-index/common -l
NAME                	CHART VERSION	APP VERSION	DESCRIPTION
bitnami-index/common	1.15.1       	1.15.1     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.14.2       	1.14.2     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.14.1       	1.13.1     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.14.0       	1.13.1     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.13.1       	1.13.1     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.13.0       	1.13.0     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.12.0       	1.12.0     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.11.3       	1.11.3     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.11.2       	1.11.2     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.11.1       	1.11.1     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.11.0       	1.11.0     	A Library Helm Chart for grouping common logic ...
bitnami-index/common	1.10.4       	1.10.0     	A Library Helm Chart for grouping common logic ...

• bitnami-pre-2022, old versions:

helm search repo bitnami-pre-2022/common -l
NAME                   	CHART VERSION	APP VERSION	DESCRIPTION
bitnami-pre-2022/common	1.14.2       	1.14.2     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.14.1       	1.13.1     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.14.0       	1.13.1     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.13.1       	1.13.1     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.13.0       	1.13.0     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.12.0       	1.12.0     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.11.3       	1.11.3     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.11.2       	1.11.2     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.11.1       	1.11.1     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.11.0       	1.11.0     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.10.4       	1.10.0     	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	1.10.3       	1.10.0     	A Library Helm Chart for grouping common logic ...
...
bitnami-pre-2022/common	0.2.4        	0.2.4      	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	0.2.3        	0.2.3      	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	0.2.2        	0.2.2      	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	0.2.1        	0.2.1      	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	0.2.0        	0.1.0      	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	0.1.1        	0.1.0      	A Library Helm Chart for grouping common logic ...
bitnami-pre-2022/common	0.1.0        	0.1.0      	A Library Helm Chart for grouping common logic ...

[spmse]

@adawalli the information you provided seems to be correct for both cases:

When first trying the workaround to add the pre-2022 repo I got the following results:

$ helm repo add bitnami-pre-2022 \
  https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami

Error: looks like 
"https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami" 
is not a valid chart repository or cannot be reached: 
error unmarshaling JSON:
  while decoding JSON: 
    invalid character 'd' looking for beginning of object key string

When retrying the exact same command at a later point in time I got this result:

"bitnami-pre-2022" has been added to your repositories

Right now we can access the required charts again, but since there was different behavior this seems a bit unstable to me 🤔 .

[jfklingler]

$ helm repo add bitnami-pre-2022 https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
"bitnami-pre-2022" has been added to your repositories

Isn't this just moving the problem from CloudFront to Github? It's a large file no matter who's serving it.

[obrunodelgado]

Oh my god, seriously that you did it?

[dunn]

Was there truly no way to negotiate with AWS on the bandwidth limits?

At the very least, as @GMZwinge said, you should be providing the latest of each historical major release.

[nodesocket]

What's going on? Just today started getting:

Error: can't get a valid version for repositories postgresql. Try changing the version constraint in Chart.yaml

Chart.yaml is:

  - name: postgresql
    version: 10.3.18
    repository: "@bitnami"

This is absurd, can't just break everything for people. Additionally I'm not going to add a helm repo pointing to a GitHub raw commit.

$ helm repo add bitnami https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami

Bitnami this is horrible developer experience.

[potiuk]

Hi @potiuk, we apologise for the inconvenience. The Helm charts you built & distributed previously still are working properly for users, the previous version of the Helm chart is included into the tarball itself.

The part that is affected is the helm package or helm dependency update actions as part of the packaging process. All the charts are publicly available, all versions. The workaround mentioned into the post allows you to continue building the Helm chart with the exact same version.

Yep. Thanks for the information - this is indeed the case., which I've learned since writing it. So the problem is indeed limited to people who are developing the charts, CI, development pipelines and generally anyone adventorous etc. Which is stil quite a big deal to get overnight change like that. This seriously undermines usefulness of using bitnami charts in the future for any serious project. But maybe that's what you wanted to achieve - this is a free offering which you can't monetize. Travis CI did similar things in the past by deprioritising free OSS support and maybe won it in the "commercial" front but lost big time (mainly to GitHub Actions) on the OSS front. That's really business not technical decision that you probably considered while making the change.

For us the decision is made - we will simply vendor your chart in. I actually even started to discuss it day before yesterday as we saw the flakiness - but now we accelerate this move.

Also one more comment and constructive suggestion. Surprises like that might easily break reputation. Maybe reputation in the free/OSS is not the most important for VMware-owned company, but I think if you want to really solve the problem, and still keep the free offreing "serious" for the free/OSS development users, you should simply deliver each chart separately as separate repository (or basically a sub-folder of the main folder that has separate index).

This is the only sustainable way to make it works long term and much more in-line with Helm 3 approach where they recognized the problem of centralized repos. You should follow their blueprint.

The cncf foundation actually made the decision about getting rid of the centralised repo and even made a deliberate effort to reach out to the community (we got plenty of discussions with cncf folks about it) to prepare everyone for the change well in advance. This change went really went and smooth and I think that was a model approach to change management. You probably know about it but for the passers-by this is just one of the blog posts where they warned about the change. https://www.cncf.io/blog/2020/10/07/important-reminder-for-all-helm-users-stable-incubator-repos-are-deprecated-and-all-images-are-changing-location/

[nodesocket]

Also, the suggested link with all versions is returning 404 as well:

https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami

Any recommendations? Need to get this resolved, breaking all of our deployments.

[thomasv314]

@nodesocket The suggested link is the repo URL you add via a helm repo command or in your declarative infra.

# This will 404 if you visit it in the browser
$ helm repo add https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/

Helm will then reach out to https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/index.yaml, which returns a 200

[wacuuu]

Now that @potiuk has opened my eyes to the monetization topic(thank you very much for that, this is something I always forget), maybe this is some crazy decision related to vmware aquisition by broadcom? This is conspiracy theory, I know, but yeah, money in the CDN mentioned in the discussion maybe?

[nodesocket]

@thomasv314 thanks. I tried the following:

$ helm repo remove bitnami
"bitnami" has been removed from your repositories

$ helm repo add bitnami https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/
"bitnami" has been added to your repositories

$ helm repo update

But now getting:

Error: no cached repository for helm-manager-54d2620bbb6f1bb3f35d4c7f945bfa25077949488dcbb0a4d01c90f2c35baa59 found. (try 'helm repo update'): open /home/ubuntu/.cache/helm/repository/helm-manager-54d2620bbb6f1bb3f35d4c7f945bfa25077949488dcbb0a4d01c90f2c35baa59-index.yaml: no such file or directory

Running Helm 3.8.2:

$ helm version
version.BuildInfo{Version:"v3.8.2", GitCommit:"6e3701edea09e5d55a8ca2aae03a68917630e91b", GitTreeState:"clean", GoVersion:"go1.17.5"}

[potiuk]

Now that @potiuk has opened my eyes to the monetization topic(thank you very much for that, this is something I always forget), maybe this is some crazy decision related to vmware aquisition by broadcom? This is conspiracy theory, I know, but yeah, money in the CDN mentioned in the discussion maybe?

I wouldn't think of it as "conspiracy" theory. I can totally relate to "free offering having no guarantees" - no complaints on it. And if that's a business decision to be less "free users" friendly, so be it - this perfectly understandable, just painful for those who put their trust in the free service (wiithout any guarantees of course). As an OSS maintainer - I can totally relate to that but while we provide no guarantees, we at least document and agree to policies that we as "free software maintainers under the Apache Software Foundation umbrella" can communicate to our users well in advance (and yeah - we also relentlessly refuse any kind of support when someone "demands" free support beyond those policies).

So this is more the matter of some kind of expectations the users have - commmunicating them and sticking to it. If they are undocumented, there are some implicit assumptions. Changing (or actually introducing) policies overnight without earlier communication and warnings which are effective immediately is one of the worst ways I can imagine to get strong, supportive community and user's trust. That's it.

Reputation can be built for years and lost overnight (especially if there is a lack of self-reflection and responding to your users).

[nodesocket]

Now, breaking on RabbitMQ, even with using helm repo add bitnami https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/.

The Chart.yaml for our RabbitMQ is:

annotations:
  category: Infrastructure
apiVersion: v2
appVersion: 3.9.8
dependencies:
  - name: common
    repository: https://charts.bitnami.com/bitnami
    tags:
      - bitnami-common
    version: 1.x.x
  - name: rabbitmq
    version: 8.24.1
    repository: https://charts.bitnami.com/bitnami
description: Open source message broker software that implements the Advanced Message Queuing Protocol (AMQP)
engine: gotpl
home: https://github.com/bitnami/charts/tree/master/bitnami/rabbitmq
icon: https://bitnami.com/assets/stacks/rabbitmq/img/rabbitmq-stack-220x234.png
keywords:
  - rabbitmq
  - message queue
  - AMQP
maintainers:
  - email: [email protected]
    name: Bitnami
name: rabbitmq
sources:
  - https://github.com/bitnami/bitnami-docker-rabbitmq
  - https://www.rabbitmq.com
version: 8.24.1

Error message is:

Error: can't get a valid version for repositories rabbitmq. Try changing the version constraint in Chart.yaml

$ helm repo list
NAME      	URL
stable    	https://charts.helm.sh/stable
minio     	https://helm.min.io
datadog   	https://helm.datadoghq.com
fusionauth	https://fusionauth.github.io/charts
bitnami   	https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/

[nodesocket]

@carrodher any idea why RabbitMQ is still broken even when using bitnami https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/?

- name: rabbitmq
    version: 8.24.1
    repository: https://charts.bitnami.com/bitnami

[cep21]

Breaking an index.yaml seems not great: it's kind of like breaking an API version without a major version change. Some alternatives

• Give each subchart their own index.yaml (charts.bitnami.com/postgres/index.yaml)
• Keep the old index.yaml, deprecate that location (no longer give it updates), and put newer charts on a different URL with different guarantees about longevity
• Reduce the size of index.yaml by removing everything not necessary (icon, annotations, created, etc)

[nodesocket]

Keep the old index.yaml, deprecate that location (no longer give it updates), and put newer charts on a different URL with different guarantees about longevity

Yes 💯 .

[aariacarterweir]

This is absolutely cooked