CI Process moving forward

[udnay]

From prafull01

Current approach: It is to build the cert signing utility, push it to the GCR repo and use that image in our tests. To implement this approach, we had to use the pull_request_target which allow forked workflows to have access to the GCR tokens.

Advatages: You have to build the image only once and use it in as many tests you want.

Disadvantages: The forked workflow (untrusted) will have access to the GCR token and will be able to push the image to our GCR repo. The multiple pushs on a PR, may be because of bug fixes, review comments etc, builds a new image will be pushed to GCR. Again waste of resources for us.

Every CI change we need have to merged before it can actually run, because pull_request_target checkout the code of the master and then we apply the pull request head ref on top of it.

Alternative Approach: Instead of building and pushing the image, we only build the image as one step in the every e2e tests job and do not push it to GCR repo. Hence workflow don't need the access to GCR repo . This build image on the same github runner can be used to run in tests. We will add a post job which triggers on merging a PR and it will initiate a docker build and push that to the GCR repo. So we will have only single image per PR merge in our GCR repo.

Advantages: The forked workflow don't need access to GCR token through secrets. Only merged PRs 1 image will be pushed to GCR repo. CI can be tested while making the change as pull_request runs directly on the pull request and if the pull request have the CI change it incorporates those changes.

Disadvantages: Might have to build the docker image multiple times, here 2 times. One for E2e for install and E2E for rotate..

/cc @rail @keith-mcclellan lets discuss here

Jira issue: HELM-5

[udnay]

I think you're right @prafull01 we should move to the alternative approach and only build the docker image for master and push that to the dev gcr.io registry. For actual releases we will push to our main docker hub image.

@rail do you have best practices we can follow?

[rail]

Another option would be publishing images from PRs to GitHub's repository. This way we can verify that we can build docker images and reuse them, while separating the main and PR docker repos.

For "official" releases (off of master or releases) we can have a separate pipeline or job to publish images to the main repo.

[prafull01]

The option to use github container registry gives us advantages of both approach.

1. You can use pull_request in the github actions. It give the Read only access to GITHUB_TOKEN to forked workflows, so forked workflow don't need to use the pull_request_target here to push to the ghcr.io
2. CI changes can directly be tested and without merging into master branch due to pull_request.
3. Can reuse the image build if we increased the E2E jobs in future.
4. ghcr.io is free for public repository, so don't have overhead of waste resources.
5. On merge, have different job to build and push the master branch image to GCR.

[prafull01]

If we go ahead with ghcr.io approach, then we need to create the GITHUB_TOKEN (Personal Access Token of the OWNER) secret added into the repo for the workflows to be able to push the image. The image name will be ghcr.io/OWNER/IMAGE

/cc @udnay

[keith-mcclellan]

@rail what are your thoughts?

[rail]

I haven't tried it, but reading the docs it sounds that you don't need to store anything in the GitHub secrets - GitHub should automatically create GITHUB_TOKEN for us. We can try it. If it doesn't work, then I'd vote to have 2 different docker repos and secrets for CI and master.