Dependency update - Golang 1.22.0

[MadhavJivrajani]

Tracking info

Link to any previous tracking issue: https://github.com/kubernetes/release/issues/3076

Golang mailing list announcement: for go1.22 https://groups.google.com/g/golang-announce/c/TpowDYVBMoY/m/3YwLkOoOAAAJ

SIG Release Slack thread: https://kubernetes.slack.com/archives/CJH2GBF7Y/p1707291164014399

Work items for Go1.22rc1

• [x] kube-cross image update: https://github.com/kubernetes/release/pull/3421

  • [x] image promotion: https://github.com/kubernetes/k8s.io/pull/6302

• [x] go-runner image update: https://github.com/kubernetes/release/pull/3421

  • [x] image promotion: https://github.com/kubernetes/k8s.io/pull/6301

• [x] releng-ci image update: https://github.com/kubernetes/release/pull/3421

  • [x] image promotion: not promoting this image

Work items for Go1.22rc2

• [x] kube-cross image update: https://github.com/kubernetes/release/pull/3433

  • [x] image promotion: https://github.com/kubernetes/k8s.io/pull/6331

• [x] go-runner image update: https://github.com/kubernetes/release/pull/3433

  • [x] image promotion: https://github.com/kubernetes/k8s.io/pull/6332

• [x] releng-ci image update: https://github.com/kubernetes/release/pull/3433

  • [x] image promotion: not promoting this image

After kube-cross image promotion

• [x] kubernetes/kubernetes update (master): https://github.com/kubernetes/kubernetes/pull/122889

  Ensure the following have been updated within the PR:

  • [x] kube-cross image
  • [x] go-runner image
  • [x] publishing bot rules
  • [x] test image
  • [x] .go-version file

Note This update may require an update to go.sum files, for example: https://github.com/kubernetes/kubernetes/pull/118507 This will require an API Review approval.

After go-runner image promotion

• [x] distroless-iptables image update: https://github.com/kubernetes/release/pull/3440

  • [x] image promotion:

After kubernetes/kubernetes (master) has been updated

• [x] k8s-cloud-builder image update: https://github.com/kubernetes/release/pull/3439

• [x] k8s-ci-builder image variants update: https://github.com/kubernetes/release/pull/3439

• [x] kubekins/krte image updates: https://github.com/kubernetes/test-infra/pull/31761

Work items for Go1.22

• [x] kube-cross image update: https://github.com/kubernetes/release/pull/3454

  • [x] image promotion: https://github.com/kubernetes/k8s.io/pull/6398

• [x] go-runner image update: https://github.com/kubernetes/release/pull/3454

  • [x] image promotion: https://github.com/kubernetes/k8s.io/pull/6399

• [x] releng-ci image update: https://github.com/kubernetes/release/pull/3454

  • [x] image promotion: https://github.com/kubernetes/k8s.io/pull/6400

After kube-cross image promotion

• [x] kubernetes/kubernetes update (master): https://github.com/kubernetes/kubernetes/pull/123217

  Ensure the following have been updated within the PR:

  • [x] kube-cross image
  • [x] go-runner image
  • [x] publishing bot rules
  • [x] test image
  • [x] .go-version file

Note This update may require an update to go.sum files, for example: https://github.com/kubernetes/kubernetes/pull/118507 This will require an API Review approval.

After go-runner image promotion

• [x] distroless-iptables image update: https://github.com/kubernetes/release/pull/3457

  • [x] image promotion: https://github.com/kubernetes/k8s.io/pull/6402

After distroless-iptables image promotion

• [x] kubernetes/kubernetes update (master): https://github.com/kubernetes/kubernetes/pull/123217

  Ensure the following have been updated within the PR:

  • [x] distroless-iptables image
  • [x] test image

After kubernetes/kubernetes (master) has been updated

• [x] k8s-cloud-builder image update: https://github.com/kubernetes/release/pull/3463

• [x] k8s-ci-builder image variants update: https://github.com/kubernetes/release/pull/3463

• [x] kubekins/krte image updates: https://github.com/kubernetes/test-infra/pull/31940

Follow-up items

• [ ] Ensure the Golang issue template is updated with any new requirements
• [ ] <Any other follow up items>

cc: @kubernetes/release-engineering cc: @liggitt

[MadhavJivrajani]

Tracking PRs related to go1.22 that we would need to pick to release branches as part of an eventual update to go1.22:

• https://github.com/kubernetes/kubernetes/pull/120750

  • 1.29: merged during 1.29 release cycle.
  • 1.28: https://github.com/kubernetes/kubernetes/pull/122580
  • 1.27: https://github.com/kubernetes/kubernetes/pull/122581
  • 1.26: EoL
  • 1.25: EoL

• https://github.com/kubernetes/kubernetes/pull/122412

  • 1.29: https://github.com/kubernetes/kubernetes/pull/122429
  • 1.28: https://github.com/kubernetes/kubernetes/pull/122428
  • 1.27: https://github.com/kubernetes/kubernetes/pull/122427
  • 1.26: EoL
  • 1.25: EoL

• https://github.com/kubernetes/kubernetes/pull/121476

  • 1.29: merged during 1.29
  • 1.28: https://github.com/kubernetes/kubernetes/pull/122580
  • 1.27: https://github.com/kubernetes/kubernetes/pull/122581

---

Tracking verification of conditions for bumping go1.21 on release branches:

• [x] ~6th May: 3 months after go 1.22 release (6th Feb 2024)
• [x] ~17th May: 1 month after Kubernetes 1.30 release on go1.22 (17th Apr 2024)
• [x] no regressions relative to go 1.22 known to impact Kubernetes
  • [x] https://github.com/golang/go/issues/65625 - Fixed in go1.22.4
• [x] pass unit and integration tests with both go 1.22 and the original go version for the release branch
  • [x] 1.27: https://github.com/kubernetes/kubernetes/pull/124921
  • [x] 1.28: https://github.com/kubernetes/kubernetes/pull/124922
  • [x] 1.29: https://github.com/kubernetes/kubernetes/pull/124923
• [x] any behavior changes in go 1.22 are mitigated to preserve existing behavior without requiring action by Kubernetes end-users
  • [ ] 1.26: NA
  • [ ] 1.27: NA
  • [ ] 1.28: NA
  • [ ] 1.29: NA

---

Tracking the PR that actually bumps to go1.22:

• [x]
  • [ ] 1.26: EoL
  • [x] 1.27: https://github.com/kubernetes/kubernetes/pull/125701
  • [x] 1.28: https://github.com/kubernetes/kubernetes/pull/125670
  • [x] 1.29: https://github.com/kubernetes/kubernetes/pull/125668

[pohly]

https://github.com/kubernetes/kubernetes/pull/120750 is getting cherry-picked in:

• 1.29: not needed (was included)
• 1.28: https://github.com/kubernetes/kubernetes/pull/122580
• 1.27: https://github.com/kubernetes/kubernetes/pull/122581

[cpanato]

/assign i will work on the release engineering side, image bumps and other things :)

[liggitt]

add https://github.com/golang/go/issues/65625 as a potential go1.22 regression to track (either until it is resolved, or until we are sure it doesn't impact Kubernetes' use of go)

[cpanato]

maybe we can close this issue and open a new one to track https://github.com/kubernetes/release/issues/3280#issuecomment-1726964825

wdyt? @liggitt @MadhavJivrajani

[MadhavJivrajani]

@cpanato I'm not sure I fully understand why we need a new issue - maybe I'm missing something?

@liggitt - tracked https://github.com/golang/go/issues/65625 as a regression, thanks!

[liggitt]

maybe we can close this issue and open a new one to track #3280 (comment)

I have a slight preference for keeping everything associated with the minor update in a single issue, but it is a little weird for an issue titled go1.22.0 to track the minor update of release branches to an eventual go1.22.x... maybe we could truncate the title to go1.22?

[cpanato]

ok not a problem, was more a suggestion :) we keep this open

[liggitt]

https://github.com/kubernetes/kubernetes/pull/121476 bumped honnef.co/go/tools to v0.4.6

it turns out that release specifically contained a fix for evaluating go1.22: https://github.com/dominikh/go-tools/releases/tag/2023.1.6

please add that to the list of updates tracked in https://github.com/kubernetes/release/issues/3280#issuecomment-1726964825 needed for go1.22

[MadhavJivrajani]

@liggitt good catch! Thanks, I've tracked the PR.

[MadhavJivrajani]

@pohly FYI ^ We'll probably need to backport https://github.com/kubernetes/kubernetes/pull/121476 to 1.27 and 1.28

[pohly]

Do we want to backport golangci-lint v1.55.1 or just bump up honnef.co/go/tools to v0.4.6 on older branches?

[liggitt]

not sure... maybe do an experimental PR to release-1.29 that bumps .go-version to 1.22.0 and just honnef.co/go/tools to v0.4.6 and see if CI passes? if so, maybe limit to just the honnef.co/go/tools bump?

[liggitt]

@liggitt good catch! Thanks, I've tracked the PR.

thanks... can go ahead and mark 1.26 as not backported for all of those because it reached EOL

[pohly]

release-1.29 is already on golangci-lint v1.55.1 and thus honnef.co/go/tools v0.4.6. release-1.28 and older aren't.

When bumping only honnef.co/go/tools v0.4.6 together with .go-version, linting passed on release-1.28 and release-1.27. Code gen wasn't happy about .go-version at 1.22.0, but I suppose that's separate.

Shall I turn those PRs into proper updates of honnef.co/go/tools?

[liggitt]

Shall I turn those PRs into proper updates of honnef.co/go/tools?

yes, that would be great

[MadhavJivrajani]

Now that we are about to start the process of bumping the Go major version on release branches, we have one tracked regression that seems to affect runc: https://github.com/golang/go/issues/65625

However, this issue is not a bug in Go itself but in glibc (specifically starting from glibc 2.25): https://github.com/golang/go/issues/65625#issuecomment-2036320298. However, the issue seems to be triggered in go1.22 because of it calling pthread_getattr_np(pthread_self(), &attr)

There is an attempt at gracefully handing this in go, however that also does not seem to be a sustainable solution: https://go-review.googlesource.com/c/go/+/585019

The workaround here needs to be in runc itself as mentioned here: https://github.com/opencontainers/runc/issues/4233

cc @liggitt

[MadhavJivrajani]

cc @akhilerm

[BenTheElder]

Though also we're pretty sure that bug only affects runc with nsenter, which kubelet is not doing, so as long as your runc binary is not with go 1.22 kubelet can be as far as I know.

[humblec]

Now that we are about to start the process of bumping the Go major version on release branches, we have one tracked regression that seems to affect runc: golang/go#65625

However, this issue is not a bug in Go itself but in glibc (specifically starting from glibc 2.25): golang/go#65625 (comment). However, the issue seems to be triggered in go1.22 because of it calling pthread_getattr_np(pthread_self(), &attr)

There is an attempt at gracefully handing this in go, however that also does not seem to be a sustainable solution: https://go-review.googlesource.com/c/go/+/585019

The workaround here needs to be in runc itself as mentioned here: opencontainers/runc#4233

cc @liggitt

The GO cherry-pick has been approved https://github.com/golang/go/issues/67650 and mostly its going to be part of Go 1.22.4.

[MadhavJivrajani]

I see an incoming Go CVE: https://github.com/golang/go/issues/67682

Which means go1.22.4 might come sooner than expected. @dims / @liggitt thoughts on waiting it out? The other alternative is we can bump release branches to go1.22.3 since the regression does not affect K8s itself. We would however have to be on a differing Go version than runc for at least one patch release, which could be acceptable.

[humblec]

I see am incoming Go CVE: golang/go#67682

Which means go1.22.4 might come sooner than expected. @dims / @liggitt thoughts on waiting it out? The other alternative is we can bump release branches to go1.22.3 since the regression does not affect K8s itself. We would however have to be on a differing Go version than runc for at least one patch release, which could be acceptable.

@MadhavJivrajani imo, considering a CVE fix is also part of 1.22.4, I think it's good to bump k/k on 1.22.4.

[dims]

The other alternative is we can bump release branches to go1.22.3 since the regression does not affect K8s itself.

My vote is to keep the wheels turning as we may catch things in CI jobs from what we have now. differing Go version than runc is not a concern i have.

[BenTheElder]

+1 @dims, we're often having a different version anyhow when we're using prebuilt upstream binaries versus our source builds ...

Unless we know that the packages kubelet uses are not supported on the go version, this should be fine.

We continue, I think we only have the codegen issue which only applies to older branches when the go image isn't also updated?

[MadhavJivrajani]

@dims @BenTheElder go1.22.4 is slated for Tuesday (4th June): https://groups.google.com/d/msgid/golang-dev/hro34FA2QbSJZ-wwA7vkTQ%40geopod-ismtpd-25?utm_medium=email&utm_source=footer

Considering we've already kicked off testing with 1.22.3 and that hasn't surfaced failing jobs (except the codegen issue), I do think 4th June is reasonable to wait for. Please lmk if you have concerns with this and I can start working towards bumping to go1.22.3. Thanks!

[dims]

@MadhavJivrajani personally i like getting things in and watch the periodic ci jobs over the weekend when there is not that much activity and then come monday there is a set of things to do .. but it's your call :) i am ok either way in this instance.

[dims]

@MadhavJivrajani are we done here?

[cpanato]

i will build the images for the release branches :)

[cpanato]

all done :) /close

[k8s-ci-robot]

@cpanato: Closing this issue.

In response to this:

all done :) /close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.