release v1beta4

[chendave]

(edited by neolit123)

general action items:

• [x] Add v1beta4 boilerplate: https://github.com/kubernetes/kubernetes/pull/118762

• [x] Add v1beta4 to kubeadm scheme but prevent the --config codepath from using it? Add --experimental flag for API migration/validate CLI commands: see https://github.com/kubernetes/kubernetes/pull/118762#discussion_r1235084133 https://github.com/kubernetes/kubernetes/pull/118866

• [x] restore TestMigrateOldConfigFromFile in common_test.go once v1beta4 is added. See how we can make this test optional if only one API version is available (e.g. in the future only v1beta4 is avaiable and v1beta3 is removed) https://github.com/kubernetes/kubernetes/pull/118941

• [x] Make dryRun a config item in the config file in v1beta4 for reset/init/join/upgrade. https://github.com/kubernetes/kubernetes/pull/119279 - initConfiguration

• [x] Loose the limitation to skip all in the init/join/... config file https://github.com/kubernetes/kubernetes/pull/113583#discussion_r1263378882 https://github.com/kubernetes/kubernetes/pull/119351

• [x] Deprecate dryRun, ignorePreflightErrors flags and others when they are supported by the all the configuration (reset, upgrade...) - https://github.com/kubernetes/kubernetes/pull/119279

• [x] merge the flag set with the configuration ignorePreflightErrors set or just override the config when flag is used // to be decided. https://github.com/kubernetes/kubernetes/pull/113583#discussion_r1263370605

• [x] Support the kubeadm config validation/migrate/print for the kubeamd reset config file, see: https://github.com/kubernetes/kubernetes/pull/113583#discussion_r1261422982 https://github.com/kubernetes/kubernetes/pull/119345 https://github.com/kubernetes/kubernetes/pull/119319 https://github.com/kubernetes/kubernetes/pull/119313

• [x] finalize v1beta4 tasks;
  • [x] document the v1beta4 changes set and prepare a release note
  • [x] move v1beta4 to the first element (highest priority) in api/kubeadm/scheme/scheme.go#AddToScheme()
  • [x] support for v1beta4 'config migrate' and 'config validate'
• https://github.com/kubernetes/kubernetes/pull/125029

• [x] Start using v1beta4 in kinder e2e tests

• https://github.com/kubernetes/kubeadm/pull/3067

---

list of features:

• [x] Support skipping addons image pull - need review / owner: @ruquanzhao https://github.com/kubernetes/kubeadm/issues/2603 https://github.com/kubernetes/kubernetes/pull/114534

• [x] Implementation of UpgradeConfiguration API types - **need review / owner: @calvin0327 ** https://github.com/kubernetes/kubeadm/issues/2489 https://github.com/kubernetes/kubernetes/pull/114062 https://github.com/kubernetes/kubernetes/pull/123068

• [x] Implementation of ResetConfiguration API types - need review / owner: @chendave https://github.com/kubernetes/kubeadm/issues/2489 https://github.com/kubernetes/kubernetes/pull/113583

• [x] Support custom env in kubeadm ControlPlaneComponent - implementable / owner: @chendave https://github.com/kubernetes/kubeadm/issues/2845 https://github.com/kubernetes/kubernetes/pull/118867

• [x] add API support for controlling various timeouts during init / join - implementable https://github.com/kubernetes/kubeadm/issues/2463 https://github.com/kubernetes/kubernetes/pull/103226 https://github.com/kubernetes/kubernetes/pull/119015 https://github.com/kubernetes/kubernetes/pull/122529 implementable / owner: @neolit123

• [x] handling of extraArgs which are allowed multiple times https://github.com/kubernetes/kubeadm/issues/1601 - status: implementable / BREAKING change. / owner: @neolit123 https://github.com/kubernetes/kubernetes/pull/119156 https://github.com/kubernetes/kubernetes/pull/125029 adds support for non-duplicate --*extra-args flags, but also deprecates them

• [ ] ~Allow overwrite KubeletRunDirectory when init/join https://github.com/kubernetes/kubeadm/issues/2104 - status: implementable / low priority / owner: @ruquanzhao~

• [x] ~graduate kubeadm patch support to GA https://github.com/kubernetes/kubeadm/issues/2046 - status: waiting for feedback~ DROPPED

• [x] Allow usage of KubeletConfiguration next to JoinConfiguration on kubeadm join https://github.com/kubernetes/kubeadm/issues/1682 - status: waiting for feedback

• [x] Switch to generating ECDSA keys https://github.com/kubernetes/kubeadm/issues/1535 - status: implementable / need agreement on the approach / BREAKING change. / owner: @neolit123 https://github.com/kubernetes/kubeadm/issues/1535 https://github.com/kubernetes/kubernetes/pull/120417 https://github.com/kubernetes/kubeadm/issues/3003 https://github.com/kubernetes/kubernetes/pull/123054

• [x] support parallel image pulls https://github.com/kubernetes/kubeadm/issues/2780 implementable / owner: @neolit123

• [x] https://github.com/kubernetes/kubernetes/pull/122616 add the support in join and init
• [x] support it in upgrade "node" and "apply"

There are already PR ready for the first three, others either need the further discussion or need someone's commitment to own.

---

BREAKING change

• handling of extraArgs which are allowed multiple times https://github.com/kubernetes/kubeadm/issues/1601 extraArgs in the ControlPlaneComponent is expected to be updated, it is currently map[string]string, it's going to be replaced by a more flexible type to allow duplicated key.

• Switch to generating ECDSA keys https://github.com/kubernetes/kubeadm/issues/1535 FG: PublicKeysECDSA will be dropped but a cluster level field (config item in ClusterConfiguration?) will be used to support the ECDSA key NOTE: this will just change the way how to support the ECDSA key, the API or the functionality will be same as before.

• some timeout fields are planned to be moved in v1beta4: https://github.com/kubernetes/kubeadm/issues/2463

---

Design is not clear at the moment

• ~Allow overwrite KubeletRunDirectory when init/join https://github.com/kubernetes/kubeadm/issues/2104 This is not clear by far, @ruquanzhao is working on a POC and provide a design if necessary.~ DROPPED

[chendave]

ref: https://github.com/kubernetes/enhancements/issues/970#issuecomment-1467580054

[chendave]

@neolit123 @SataQiu @pacoxu would like to hear your feedback on this. Is this list enough for v1beta4?

[chendave]

ref: https://github.com/kubernetes/kubernetes/pull/114534#issuecomment-1584268650

[chendave]

/assign

[neolit123]

please include all labeled issues https://github.com/kubernetes/kubeadm/labels/kind%2Fapi-change (even if some are a bit vague / not actionable)

[neolit123]

i've added this to .29. i think if this api code work is to happen it should be at the start of a release cycle. but perhaps we can get all the planning done in .28. what do people think?

[chendave]

please include all labeled issues

I hope I haven't miss anything, the list is updated.

(even if some are a bit vague / not actionable)

Haven't got the chance to go through each of them, but I will evaluate each of them in the near future.

[chendave]

i think if this api code work is to happen it should be at the start of a release cycle. but perhaps we can get all the planning done in .28

+1. we can also get some of them implemented within the time frame of .28 after the scope if finalized.

[neolit123]

i was thinking that if we want to work on an api for multiple releases, we could try this:

• add the new api boilerplate / clone v1beta3
• prevent the users from using it via cli / wip errors in a few key places
• add all new api changes
• remove the wip errors
• release the api

the tricky part would be getting kubeadm to agree on this wip status.

in this approach, a single release note must be used listing all changes when we are ready - i.e. all api changes must go without release notes.

reminder - we should not iterate new features over a released api, because it causes pain for consumers like capi. ideally, an api should be a released, locked feature set.

[chendave]

add the new api boilerplate / clone v1beta3 prevent the users from using it via cli / wip errors in a few key places

+1, I believe we can start this in .28 release, and part of features could be added in .28 as well, otherwise, it would be a little rush for .29 release. We will definitely has some warning for v1beta4 before the official release.

[pacoxu]

reminder - we should not iterate new features over a released api, because it causes pain for consumers like capi. ideally, an api should be a released, locked feature set.

Should we include the CAPI team and other consumers in this topic to get more feedback?

[neolit123]

reminder - we should not iterate new features over a released api, because it causes pain for consumers like capi. ideally, an api should be a released, locked feature set.

Should we include the CAPI team and other consumers in this topic to get more feedback?

we could email the sig mailing list.

[chendave]

@neolit123 @pacoxu I am currently triage all those requirements, will send a summarization to the mailing list or join the sig community meeting for the head-ups and ask for review comments.

[ruquanzhao]

/cc

[SataQiu]

work on an api for multiple releases

We can introduce the v1beta4 API, but not register it into the Scheme, prevent users to use it. https://github.com/kubernetes/kubernetes/blob/18d05b646d09b2971dc5400bc288062b0414e8cf/cmd/kubeadm/app/apis/kubeadm/scheme/scheme.go#L42-L46

[chendave]

We can introduce the v1beta4 API, but not register it into the Scheme, prevent users to use it.

that's indeed a good suggestion, thanks @SataQiu !

[neolit123]

FYI. in yesterday's Sig meeting we discussed kubeadm's v1beta4 plan. @fabriziopandini requested that we tag in the above list what is a [BREAKING] change. one example is allowing duplicate extraArgs. please check if there is anything else.

https://docs.google.com/document/d/1Gmc7LyCIL_148a9Tft7pdhdee0NBHdOfHS1SAF0duI4/edit?usp=drivesdk

we also discussed whether this should be a v1 or a v1beta4. my vote was to gather motivation from the kubelet which is moving to v1. perhaps v1beta4 should be our last beta. ideally, this means we should do all the planned breaking changes in this beta4. otherwise we need to plan v2 after the v1.

[chendave]

@neolit123 thanks for bringing this to the sig meeting!

we tag in the above list what is a [BREAKING] change

sure, will do.

[fabriziopandini]

thanks for adding [breaking]; it will echo yesterday discussion today at the CAPI office hours and report back feedback/ask people to provide feedback directly here

[neolit123]

i can try sending the boilerplate pr for v1beta4 later today. copy v3 -> v4, essentially.

work on an api for multiple releases

We can introduce the v1beta4 API, but not register it into the Scheme, prevent users to use it. https://github.com/kubernetes/kubernetes/blob/18d05b646d09b2971dc5400bc288062b0414e8cf/cmd/kubeadm/app/apis/kubeadm/scheme/scheme.go#L42-L46

i was thinking about this. it can work partially as long as we make the right changes in the internal api. but it will not work for breaking changes e.g. removing the ecdsa FG.

we may have to add the api changes first, keep the api out of the scheme, think when to add business logic, once ready for release add the api in the scheme, but drop api changes without business logic.

[chendave]

we may have to add the api changes first, keep the api out of the scheme, think when to add business logic, once ready for release add the api in the scheme, but drop api changes without business logic.

SGTM, we can have the boilerplate as the first commit and the api change as an incremental commit.

[neolit123]

boilerplate PR https://github.com/kubernetes/kubernetes/pull/118762

[neolit123]

@ruquanzhao since you have this task:

Allow overwrite KubeletRunDirectory when init/join Allow overwrite KubeletRunDirectory when init/join #2104

i will take the other one assigned to you:

handling of extraArgs which are allowed multiple times

[pacoxu]

Too many tasks are tracked here. Should we use a simple GitHub board to track v1beta4-related things?

https://github.com/orgs/kubernetes/projects

[chendave]

thanks @pacoxu for the info! I am +1 to create the board.

Most important features which need some attention now is:

• ~support for structured ExtraArgs: https://github.com/kubernetes/kubernetes/pull/119156 (need review)~

• upgrade API type: https://github.com/kubernetes/kubernetes/pull/114062 (need some eyes, no review comments yet)

• support for controlling various timeouts during init / join:

  • https://docs.google.com/document/d/1VNGpGWb-vqblfZQLp9DY1lx777rwhwIFAjDFQNIwWh0 (design doc, I am pretty worried about this, we haven't finalized the way we should go)
  • https://github.com/kubernetes/kubernetes/pull/119015

• skipping addons image pull: https://github.com/kubernetes/kubernetes/pull/114534 (waiting for rebase and review, this should be fine as it is not a big change)

Anything else should not hard to address, e.g. enable the v1beta4 in code base, ignore the errors should be align with other flag/config etc.

[pacoxu]

https://github.com/kubernetes/kubernetes/pull/125029 is merged for v1.31.

• https://github.com/kubernetes/kubeadm/pull/3067 is WIP.

[neolit123]

closing this one. thanks everyone. let's track bugs / feedback in separate new tickets.