Release 7.12.0

[taylorsilva]

Creating this issue to discuss making a new release since I saw Rui make the milestone for one.

Are there any blockers at this point? @xtremerui (not sure if there's anyone else on the Broadcom side I should tag too)

Steps for a new major/minor release:

• [x] Add this issue to the v<M.m.x> milestone

• [ ] Bump the appropriate versions for resource types. Go through the list of resource types below and take a look at which resource type repositories have had new commits or PRs by comparing the last release to the master branch. Take a look at what those changes entail and bump the version in their respective pipeline in ci.concourse-ci.org.

  • If the changes were only README or repo restructuring changes with no user impact, you don't need to bump the version

  • If the changes were small bug fixes or changes, you can do a patch version bump

  • If the changes were adding of features, you can do a minor version bump

  • If the changes involve a breaking changes, that should be a major version bump

  • [x] git: https://github.com/concourse/git-resource/compare/v...master

  • [ ] registry-image: https://github.com/concourse/registry-image-resource/compare/v...master

  • [x] s3: https://github.com/concourse/s3-resource/compare/v...master

  • [x] time: https://github.com/concourse/time-resource/compare/v...master

  • [x] semver: https://github.com/concourse/semver-resource/compare/v...master

  • [x] pool: https://github.com/concourse/pool-resource/compare/v...master

  • [x] mock: https://github.com/concourse/mock-resource/compare/v...master

  • [x] github-release: https://github.com/concourse/github-release-resource/compare/v...master

  • [x] hg: https://github.com/concourse/hg-resource/compare/v...master

  • [x] docker-image: https://github.com/concourse/docker-image-resource/compare/v...master

  • [x] bosh-io-stemcell: https://github.com/concourse/bosh-io-stemcell-resource/compare/v...master

  • [x] bosh-io-release: https://github.com/concourse/bosh-io-release-resource/compare/v...master

  • [x] ~tracker: https://github.com/concourse/tracker-resource/compare/v...master~ Repo archived since Tracker is EOL

• [ ] Create your release branch on the concourse/concourse github repo with the following format release/M.m.x (M being the major version and m being the minor version)

• [ ] Create the release branch on concourse/concourse-docker repository.

• [ ] Create the release branch on the concourse/concourse-bosh-release repository. Make any missing changes to the spec of web or worker depending on if the release contains any changes that adds or modifies any flags.

  • Any changes you make on the branch will not get automatically merged back to master so try to make the changes on master and then create the branch from there.
  • We should really have something that will merge the branch back to master. (like we do for the concourse/concourse branches)

• [ ] Create the release branch on concourse/concourse-bosh-deployment repository.

• [ ] Create the release branch on concourse/concourse-chart repository from the dev branch. Make any missing changes to values.yaml or templates/web-deployment.yaml for changes to flags on web or templates/worker-deployment.yaml for changes to flags on the worker.

• [ ] Add your release pipeline to the reconfigure-pipeline

• [ ] Go through all the needs-documentation PRs in the release page for your milestone https://project.concourse-ci.org/releases/concourse?milestone=v<M.m.p> and make sure that everything has proper documentation within concourse/docs (if needed). You can organize which PRs by clicking on the button to add whichever label best fits that PR.

  • If it is already documented within concourse/docs, add a release/documented label
  • If there is no documentation and the changes have user impact that should be documented, add the documentation to concourse/docs(or delegate) then add a release/documented label after finished. E.g. the addition of a new step type ( set_pipeline step).
  • If there is no documentation and the changes have user impact that do not need to be documented, add a release/undocumented label. E.g. an experimental feature.
  • If there is no documentation and the changes do not have user impact, add a release/no-impact label. E.g. refactors.

• [ ] Once the all source code changes are finalized, Concourse RC version should be deployed to CI

  • including all the external workers (pr-worker, ci-topgun-worker, darwin-worker & BOSH deployed windows worker)

• [ ] If you are doing a major release (or a release that involves a risky large feature), consider creating a drills environment for some stress testing to ensure that the release does not involve any performance regressions.

• [ ] Once the final commit has made it through the pipeline, the create-draft-release job can be triggered. This job will create a draft release within the concourse GitHub release page where you can make any final adjustments or arrangements to the generated release notes. PLEASE NOTE that any manual changes made on the draft release WILL BE OVERWRITTEN if you retrigger the create-draft-release job. Please be sure to only make manual edits AFTER you are sure this is the final run of the job.

  • If you would like to edit the content, you can directly edit the PRs that it was generated from. The title is used for each PR and also the body within the Release Note header in the PR. After you have made your edits within the PR, you can rerun the create-draft-release job in order to regenerate a new release note.
  • If you would like to change the arrangement of the PRs within the release note, you can make the edits directly on the release note of the draft release.

• [ ] Once everything is ready, the shipit job can be triggered. The publish-binaries job will convert your draft release into a final release including the body of your draft release (which will persist any changes you made to the draft release body). Subsequently, the promote concourse job will run automatically. The publish-docs job runs automatically.

• [ ] The helm-chart pipeline is used to bump & then publish the chart.

  • First, run the merge-dev-into-master job | I believe this job no longer exists and we must manually do this ourselves, probably to fix any conflicts?
  • Next, run the concourse-app-bump job (bumps the app version and image to point to the latest release)
  • Finally, run the publish-chart-{major|minor|patch} job, depending on what has changed in the chart
  • If you make a major bump, be sure to update the CHANGELOG.md in the concourse-chart repo

• [ ] Once the Concourse release is shipped, Concourse should be deployed to Hush-House

[xtremerui]

@taylorsilva thank you for creating this issue. We have an epic for this release. Whoever got assigned this issue will do the release.

[taylorsilva]

Sorry to folks seeing this issue still open. Haven't had time to start cutting the release. You know, life.

[marco-m-pix4d]

@taylorsilva @xtremerui is there any way external people like me can help?

[wayneadams]

Hi @marco-m-pix4d , thanks for the ping and sorry for the delay. Last week, we started to work though the steps in the issue template, but we needed to put it down temporarily for other priorities. The plan is that we (I) pick this up again early next week.

[marco-m-pix4d]

@wayneadams thanks a lot for the update, appreciated!

[taylorsilva]

@xtremerui Looks like Vaults' cert just expired

[taylorsilva]

Chipping away at getting a new cluster setup: https://ci.concourse-oss.org/ (it's setup with Github auth to the concourse org)

Next step is updating the pipelines. Planning to remove all the bosh jobs from the release pipeline because it's too heavy to keep maintaining. Focus will be on getting the release pipelines setup and running and pushing 7.12.0 out the door

---

For those out of the loop, the infrastructure at https://ci.concourse-ci.org/ is maintained by Broadcom. Only Broadcom employees have access to the GCP account where all the infrastructure is running. They currently have no ETA as to when they'll be able to get it fixed.

I've decided to go ahead and setup a new Concourse cluster for the community to start releasing from. Goal is to get us back on track and releasing regularly again. Once 7.12.0 is out I'm going to focus on getting the PRs pipeline working.

[marco-m-pix4d]

@taylorsilva for sure I am happy and what to thank you. But, I also want to say one thing: please be careful not to step into the slope towards open source maintainer burn out! In case of doubt, protect yourself. 💪

Also if I may ask: the infra will have a cost, who is going to fit the bill? Maybe it would be worthwhile to enable GitHub sponsorship ?

[taylorsilva]

please be careful not to step into the slope towards open source maintainer burn out! In case of doubt, protect yourself. 💪

Thanks for the reminder. I am constantly aware of this as I've read many of the same maintainer burnout stories that you likely have. I'm doing my best to not push myself very hard and just do one small thing at a time.

For infra cost, I'm footing it right now. I'm in a good financial position that I can do this, though I am trying to stay thrifty to stretch my dollars! I do want to start doing Github sponsors or OpenCollective. I'll do that once a release is out and folks can see that we are able to keep pushing releases out. I want people to feel confident giving money to the project and I think that means seeing some results first in the form of a new release.

[suhlig]

GitHub sponsors would be nice as it would give us a way to help ensure the future of Concourse.

[taylorsilva]

A little update for everyone because I'm going on vacation so won't be working on this over the next week and a bit.

I've got the main pipelines setup and running, but not everything is passing yet. I'm focusing on the main concourse pipeline right now. I want to get that pipeline fully green before making a release/7.12.0 pipeline. I also need to get all the resource-type pipelines green too.

So order right now is:

• https://ci.concourse-oss.org/teams/main/pipelines/concourse?group=all
• https://ci.concourse-oss.org/?search=team%3A%22main%22%20group%3A%22resource%22
• https://ci.concourse-oss.org/?search=team%3A%22main%22%20group%3A%22release%22

Most of my time has been spent getting tests to pass on the new infrastructure. The workers are running on Fedora 40 which only has cgroups v2 enabled. I only ran into issues with running container-in-container tests (like DinD). There's also a test failing right now that I'm pretty sure is due to Hetzner's NIC having a max MTU of 1450 instead of 1500 (see here).

Here's a snapshot of all the jobs passing right now:

I've still got to get some infra setup like an S3 bucket for storing some artifacts and for testing the S3 resource-type. I also need to modify a bunch of the k8s jobs to do one of the following:

• Run a small k8s (k3s or minikube?) on the worker and run tests against that. I'm worried that'll be a very slow test though and possibly flakey because it might overload the workers CPU
• Create a k8s cluster for the duration of the job/test (thinking Linode as they have the fastest startup times for a k8s cluster) and run the tests against that cluster. Likely to be the most robust option and the one I'll likely go with. It'll mean more $$$ but I think it'll be worth it.