Release Pulsar Helm Chart in GitHub Pages flavor

[tisonkun]

Motivation

Currently, we host Pulsar Helm Chart's metadata (index.yaml) on https://pulsar.apache.org/charts/index.yaml. Users encountered an issue today (https://github.com/apache/pulsar/issues/17799) due to we removed the file during "clean up" content before building the official website fully.

Depending on another repo is brittle. It's hard to ensure further development happening in the pulsar-site repo is aware of this trick while this is critical to the pulsar-helm-chart usage.

Proposal

I propose we adopt releasing Pulsar Helm Chart in GitHub Pages flavor to make this chart self-contained.

• Helm official document: https://helm.sh/docs/howto/chart_releaser_action/
• Chart release action: https://github.com/helm/chart-releaser-action
• A live example from OpenSearch:
  • https://github.com/opensearch-project/helm-charts/blob/main/.github/workflows/release.yaml
  • https://github.com/opensearch-project/helm-charts/blob/gh-pages/index.yaml
• Prototype on fork: https://github.com/tisonkun/pulsar-helm-chart

Implementation

The implementation steps are:

1. Create gh-pages branch on pulsar-helm-chart repo, add current index.yaml file to the branch.
2. Ask Apache INFRA team to config the branch as the GitHub Pages branch.
3. Add the workflow using chart-releaser-action (bundled in this repo, it's lightweight).
4. Redirect https://pulsar.apache.org/charts to the new place.

Risks

~~The last step may be optional because IIUC helm chart's index is global. Thus once we publish the first new index with chart-releaser-action, all downloads to the chart will find the index from the new place. Although, I'm not an expert on Helm Chart so here requires some verification.~~ No. https://pulsar.apache.org/charts is actually documented at https://pulsar.apache.org/docs/helm-overview/#upgrading. So we should make the redirection.

Another risk is that the reliability of pulsar.apache.org is promised by the ASF INFRA team, while GitHub Pages are handled by the GitHub team. I'd say GitHub is less stable than ASF sites.

Alternative

Perform an Apache release and upload the index to https://dist.apache.org/repos/dist/release/pulsar. This approach requires modifying homemade automation (dist.a.o uses SVN), and possibly brings new bugs.

[tisonkun]

cc @lhotari @mattisonchao @yaalsn @ericsyh

[michaeljmarshall]

Thanks for your proposal @tisonkun. Here is a relevant mailing list discussion https://lists.apache.org/thread/2xmks3zzkh5cm4ghrqt4ys6zwh4c3gbj.

I think we should update the release process so that it properly follows Apache guidelines, requiring signed tarballs and 3 PMC votes. I also think we should host the index.yaml and the tarballs on https://dist.apache.org/repos/dist/release/pulsar.

[ericsyh]

It makes sense to maintain the chart release in its own repo.
If possible, I think ASF Infra can consider maintaining a https://github.com/helm/chartmuseum server and all charts under ASF can use this chartmuseum server to release and manage releases.

[yaalsn]

IMO, serving index.yaml in GitHub page makes sense.

1. This file doesn't need to be visited frequently.
2. Even if GitHub turn down, Github Engs will fix it as soon as possible.
3. SN uses the GitHub page to serve the file and it works well.

[tisonkun]

Thanks for your feedback @michaeljmarshall @ericsyh @yaalsn. Sorry for the late reply.

Although any Apache TLP needs not to follow approaches adopted by other projects, I try to see how other projects do it:

CouchDB, APISIX, and ShenYu use the GitHub Actions + GitHub Pages way.

• https://github.com/apache/couchdb-helm/blob/main/.github/workflows/chart-releaser.yaml
• https://github.com/apache/apisix-helm-chart/blob/master/.github/workflows/release.yaml
• https://github.com/apache/shenyu-helm-chart/blob/main/.github/workflows/release.yml

SkyWalking and Tika publish the Chart to https://apache.jfrog.io. I don't find the release manual yet.

• https://github.com/apache/skywalking-kubernetes
• https://github.com/apache/tika-helm

@michaeljmarshall I ever thought of the release process based on dist.a.o. But thinking once more about the archive policy and upload process I tend to avoid them if we can.

Thus, I'd like to continue to introduce the GitHub Pages flavor release. @michaeljmarshall does it make sense to you?

Related actions have been added to whitelist: https://issues.apache.org/jira/browse/INFRA-23716

[tisonkun]

I'm going to submit a pull request to this repo. And it still needs a committer to help create the gh-pages branch (identical to master is OK for initialization).

UPDATE - So here is https://github.com/apache/pulsar-helm-chart/pull/292.

[michaeljmarshall]

I'm just seeing your response and PR now. Thanks for working on this @tisonkun.

I think we should get specific permission from the ASF to verify this release process. I am going to figure out how to get a discussion going to move this forward.

My primary concerns are that this flow releases code/binaries that not voted on by the PMC, release artifacts are not signed, and release artifacts are not verified before publication. This is my primary source https://www.apache.org/legal/release-policy.html.

[michaeljmarshall]

I found this relevant jira ticket: https://issues.apache.org/jira/browse/LEGAL-573. It shows that the Apache Airflow project is using a formal voting process and is having their binaries hosted on dist.apache.org. Here is one of their release votes: https://lists.apache.org/thread/cfr5q9jccdw1l61yg1qojt1po96cb8c5.

[michaeljmarshall]

The airflow process is defined here https://github.com/apache/airflow/blob/main/dev/README_RELEASE_HELM_CHART.md. I will take a closer look at it and see if we can learn from their process.

[michaeljmarshall]

Another risk is that the reliability of pulsar.apache.org is promised by the ASF INFRA team, while GitHub Pages are handled by the GitHub team. I'd say GitHub is less stable than ASF sites.

I do agree that it is risky. My preference is to host the index.yaml ourselves, but we could also start a discussion on the mailing list to see if others are opinionated.

[tisonkun]

@michaeljmarshall I've updated the title and gonna close it once we don't have a dependency on the existence of the very file (content/charts/index.yaml) - redirection is OK.

[michaeljmarshall]

Thanks @tisonkun! That sounds great.

[michaeljmarshall]

@tisonkun - I think we might need to figure out a different way to maintain the index.yaml. @dave2wave mentioned that artifacts at dist.a.o need to be signed, and given that the file would keep moving, I'm not sure how we would make it work correctly. In order to not block the 3.0.0 release, I went ahead and added the index.yaml to the main branch in pulsar-site so that it'll be slightly harder to accidentally delete.

[tisonkun]

@michaeljmarshall Thanks for your update. I noticed https://github.com/apache/pulsar-site/commit/460ae87cb24723757e768a77388c15726ed2ecf9 days ago. As long as the release process writes to update the file in the main branch instead of the deploy branch, I'm OK that it's enough to prevent accidental removal.

Closed as done.