Upstream or remove Kubernetes patches

[erikwilson]

Is your feature request related to a problem? Please describe. Audit of our k8s patch set and work to reduce that. Enumerates each change, what it does, and why it is needed.

Additional context rancher/k3s#548

Generated from:

cd ~/go/src/github.com/kubernetes/kubernetes
K8S=v1.22.2; git log --name-only --reverse --pretty=format:'* **[`%h`](https://github.com/k3s-io/kubernetes/commit/%H) %s**%n  by %an <%ae> on `%as`%n  %n%w(,2,2)%b' ${K8S}-k3s1~1...${K8S} | sed -E 's|^[[:alnum:]]+.*|    * `&`|'

Changes

• 0bc428a57b8 kubelet: new cgroup driver: "none" by Akihiro Suda [email protected] on 2019-06-02

  The "none" driver is used for running "rootless" mode on a host that does not support cgroup v2.

  This commit is specific to Usernetes and isn't going to be proposed to the Kubernetes upstream. So, there is no FeatureFlag for gating this driver.

  Signed-off-by: Akihiro Suda [email protected]

  • cmd/kubelet/app/options/options.go
  • cmd/kubelet/app/server.go
  • pkg/kubelet/apis/config/types.go
  • pkg/kubelet/cm/cgroup_manager_linux.go

• 018a084aaa8 Drop client-go cloud auth by Darren Shepherd [email protected] on 2019-09-05

  • cmd/kubectl/kubectl.go
  • pkg/kubeapiserver/authenticator/config.go

• 094c8c4758c Drop storage plugins by Darren Shepherd [email protected] on 2019-09-05

  • cmd/kube-controller-manager/app/plugins.go
  • cmd/kubelet/app/plugins.go

• 96b01267556 Drop credential providers by Darren Shepherd [email protected] on 2019-09-05

  • cmd/kubelet/app/options/globalflags.go

• f5b3ff458c7 If you can't set hashsize on nf_conntrack don't fail by Darren Shepherd [email protected] on 2018-10-05

  • cmd/kube-proxy/app/conntrack.go

• ac21553e5a5 only use the resolved name if port was zero by Darren Shepherd [email protected] on 2018-12-31

  • pkg/kubelet/cri/streaming/server.go

• 7b0c45ae378 Make kubelet.sock path changable by Darren Shepherd [email protected] on 2019-01-09

  • staging/src/k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1/constants.go

• 039ffbb1cbf Wait for kube-apiserver for 2 minutes for slow (ARM) systems by Darren Shepherd [email protected] on 2019-01-22

  • cmd/kube-controller-manager/app/controllermanager.go

• 91eb20a1943 Don't check for cpuset cgroup, not always required? by Darren Shepherd [email protected] on 2019-01-22

  • pkg/kubelet/cm/container_manager_linux.go

• 3ec6171e307 Set all sources so node+agent in the same process doesn't get restricted by Darren Shepherd [email protected] on 2019-02-07

  • cmd/kube-apiserver/app/server.go

• 38b8e7ff661 Hide deprecated warnings by Darren Shepherd [email protected] on 2019-02-07

  • cmd/kubelet/app/options/options.go
  • staging/src/k8s.io/apiserver/pkg/server/options/deprecated_insecure_serving.go

• 5e319e3a223 Add ability to disable proxy hostname check by Darren Shepherd [email protected] on 2019-02-07

  • pkg/proxy/util/utils.go

• f5172425969 Cache loopback cert in the certs dir if set by Darren Shepherd [email protected] on 2019-02-07

  • staging/src/k8s.io/apiserver/pkg/server/options/serving_with_loopback.go

• cdb18d010cc Don't ever select the flannel bridge or cni bridge by Darren Shepherd [email protected] on 2019-02-07

  • staging/src/k8s.io/apimachinery/pkg/util/net/interface.go

• 32b7b2050e1 Update kubernetes service on start for port changes by Erik Wilson [email protected] on 2019-06-21

  • pkg/controlplane/controller.go

• ec39217a8e5 Add WrappedRoundTripper() to tokenSourceTransport by Darren Shepherd [email protected] on 2019-02-27

  • staging/src/k8s.io/client-go/transport/token_source.go

• 29d1cbd8bb0 Add stopCh to apiserver & context to kublet commands by Darren Shepherd [email protected] on 2019-08-26

  • cmd/genkubedocs/gen_kube_docs.go
  • cmd/genman/gen_kube_man.go
  • cmd/kube-apiserver/apiserver.go
  • cmd/kube-apiserver/app/server.go
  • cmd/kubelet/app/server.go
  • cmd/kubelet/kubelet.go

• 5cbb88fd4c8 Allow override of kubeconfig dialer by Darren Shepherd [email protected] on 2018-10-09

  • cmd/kube-apiserver/app/server.go

• 22f3ec25de3 Notify startup to grab a hold of handler and authenticator by Darren Shepherd [email protected] on 2019-09-18

  • cmd/kube-apiserver/app/server.go

• 4759b2b71cb Ignore proxy settings for kubelet client by Erik Wilson [email protected] on 2019-04-15

  Proxy settings may interfere with a kubelet client communicating with the API server, so set the proxy to nil.

  • cmd/kube-apiserver/app/server.go
  • pkg/kubelet/client/kubelet_client.go

• cb65816a90f Allow override of "kubernetes" endpoint port by Darren Shepherd [email protected] on 2018-10-08

  • staging/src/k8s.io/apiserver/pkg/server/config.go
  • staging/src/k8s.io/apiserver/pkg/server/options/serving.go

• f8fe3d4256b Don't check etcd if scheme is unix by Darren Shepherd [email protected] on 2019-08-28

  • pkg/registry/core/rest/storage_core.go

• 39523fdeb00 Fix CSI initialization conflict by Darren Shepherd [email protected] on 2019-08-30

  CSI is used by both the kubelet and kube-controller-manager. Both components will initialize the csiPlugin with different VolumeHost objects. The csiPlugin will then assign a global variable for the node info manager. It is then possible that the kubelet gets the credentials of the kube-controller-manager and that will cause CSI to fail.

  • pkg/volume/csi/csi_plugin.go

• df45a2cbc63 Fix inconsistent etcd read by Darren Shepherd [email protected] on 2019-11-18

  • staging/src/k8s.io/apiserver/pkg/storage/etcd3/store.go

• 921e7df2305 Add openapi generator by Darren Shepherd [email protected] on 2019-08-27

  • pkg/generated/openapi/gen/main.go

• cb4f3b09912 Add tag.sh script by Darren Shepherd [email protected] on 2019-08-27

  • tag.sh

• c4ebe1924cb Add Vagrantfile by Erik Wilson [email protected] on 2019-09-19

  • Vagrantfile

[olljanat]

I ended up here as searched about k3s + rooless stuff so FYI that there is already PR open to upstream rootless patches https://github.com/kubernetes/kubernetes/pull/92863

[dims]

@erikwilson are there things here that could get into k8s 1.22? Is there a better place to look at how the upstreaming efforts are going on?

[dweomer]

@erikwilson are there things here that could get into k8s 1.22? Is there a better place to look at how the upstreaming efforts are going on?

This is probably as good a place to discuss any, prior to moving to a KEP or simple PR.

[cjellick]

Hey @dims - thanks for asking about this.

We'll work through this list in the next couple weeks to see what if anything can be moved forward in the 1.22 timeframe. Balanced against other obligations, that is somewhat tight, so we might be looking further out for bigger changes.

Is there anything in particular that you are looking for or just interested in this generally?

[dims]

@cjellick i just want the teams to start talking and doing things together :)

[AkihiroSuda]

232effd3d57 kubelet/cm: ignore sysctl error when running in userns 125fe9d9a5c kube-proxy: allow running in userns c0d770b90bf kubelet: new feature gate: Rootless

Merged in v1.22 (feature gate was renamed to KubeletInUserNamespace) https://github.com/kubernetes/kubernetes/pull/92863/commits

[brandond]

We'll be dropping a few patches in 1.22, as discussed over at https://github.com/k3s-io/k3s/issues/3596#issuecomment-876173892.

@AkihiroSuda for the CgroupNone patch, do you think we should drop that as well? I believe the effect of that would be to require delegated cgroupv2 under systemd for rootless, which means not being able to run rootless K3s from the command line?

[AkihiroSuda]

for the CgroupNone patch, do you think we should drop that as well?

Yes, not sure it should be in 1.22 or 1.23, though.

I believe the effect of that would be to require delegated cgroupv2 under systemd for rootless, which means not being able to run rootless K3s from the command line?

systemd-run --user -p Delegate=yes --tty k3s server --rootless can continue to work.

https://github.com/k3s-io/k3s/blob/238dc2086e94a06b2e625bd46473ec24ba67eb86/k3s-rootless.service#L28

[brandond]

I'm hijacking the first comment to start running down the current (as of 1.22.2) list of patches we're carrying. @dims would you be interested in going over these with me at some point to discuss which ones you think we might be able to adapt for inclusion upstream?

[dims]

why yes of course @brandond . i can't speak for all the sigs for sure and we will need to go through regular process(es) here, but yes happy to chat!

[caroline-suse-rancher]

@brandond is this safe to close?

[brandond]

no, this is an ongoing effort. We still require a handful of the above-listed patches to embed Kubernetes in K3s.

[dims]

@brandond @caroline-suse-rancher what does 2023 confirmed label mean?

[brandond]

That we're going to continue working on this in 2023.

[dims]

cc @OrlinVasilev