How to overwrite resource config for kube-apiserver

[krish7919]

I am new to KinD, and want to overwrite my kube-apiserver configs to assign different values for CPI/memory.. How do I do that..? Googling tells me I need to edit my ClusterConfiguration, but not sure as the only option supported is apiServer.extraArgs and I see no options for apiserver.resources or such?

CPU seems to be hardcoded in kubeadm to 250m: https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/controlplane/manifests.go#L66

I have also created a patch file as:

spec:
  containers:
    - name: kube-apiserver
      resources:
        requests:
          cpu: 500m
        limits:
          cpu: 500

and written it to a file and mounted it in the control plane container. Then I do this in the kind config file:

- role: control-plane
    kubeadmConfigPatches:
      - |
        kind: JoinConfiguration
        patches:
          directory: /etc/kubernetes/manifests/patches
...

However the configs are never picked up.

The third approach I tried was contacting developers/community on the slack channel and there was no response.

What would you like to be documented:

Please provide an example of how to update the resource configuration in the static pod manifests generated by kubeadm when running kind create cluster command.

Why is this needed:

This will help overwrite the default value(s) hardcoded in code when running tests in Kind (or for other use cases).

[Perhurb]

I have the same requirement. But I couldn't find any configuration for each node resource. Do you know it should be configuerd by kubeadm? Or, could you tell me if you have solute it?

[krish7919]

I have not solved it, and it seems to be hidden underneath many layers. Again, I have gone through source code, docs, googled for things, and came up with nothing over here or on Slack.

[aojea]

Those manifests are generated by kubeadm cc: @neolit123

[neolit123]

However the configs are never picked up.

~you probably have to also mount the patches on the kind nodes?~

NVM, you mention that you mounted the patches:

and written it to a file and mounted it in the control plane container. Then I do this in the kind config file:

but yes kubeadm patches can be used to patch the static pod manifest resources: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/control-plane-flags/#patches

we have unit tests / e2e tests that are green, so kubeadm patches work.

[Perhurb]

How can I write the a specified patch file for my control-plane? Is there a complete example or some relative docs?

[krish7919]

we have unit tests / e2e tests that are green, so kubeadm patches work.

That's the reason why this is tagged as documentation; I tried a bunch of things and it didn't work, and an example of what you say would be helpful. Also, I would need to retrofit it to kind workflow with it configuration, but that's a different problem to solve.

[neolit123]

I haven't tried kubeadm patches with kind. But based on prior discussions i am assuming they worked. Kubeadm would print log entries starting with [patches] .... related to this.

[BenTheElder]

This is a relatively new kubeadm feature that we haven't decided how to support or leverage yet.

KIND may be writing these patches in a future version to enable e.g. per-node kubelet config patching. We've discussed kubeadmConfigPatches => kubernetesConfigPatches and a more sophisticated patch runtime.

Patching kubeadm to this level is a deeply power user feature, and will vary by Kubernetes version.

It would be helpful if you shared your attempted config in full detail, the approach looks right but we can't see all the details here yet.

Tentatively:

I see your config snippet is a control-plane node, but you're using JoinConfiguration, which looks like a misunderstanding of kubeadm. JoinConfiguration is only used for nodes other than the first node. On the first role: control-plane node we do kubeadm init, on subsequent nodes we do kubeadm join (control-plane or otherwise). The nodes that use init use InitConfiguration and nodes that join use JoinConfiguration.

https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta3/

The third approach I tried was contacting developers/community on the slack channel and there was no response.

Sorry, Antonio and I each have a lot going on just this moment. Otherwise I would guess community members are not doing this yet. Most things in kubeadm can be configured without kubeadm's patches functionality.

[krish7919]

It would be helpful if you shared your attempted config in full detail, the approach looks right but we can't see all the details here yet.

This is my full configuration:

# Overwrite kube-apiserver config to set custom CPU limit
kube_apiserver_patch = """
spec:
  containers:
    - name: kube-apiserver
      resources:
        requests:
          cpu: 500m
        limits:
          cpu: 500
"""
            kube_apiserver_patch_file = tempfile.NamedTemporaryFile(dir="/tmp/", delete=False)
            with open(kube_apiserver_patch_file.name, "w") as fh:
                fh.write(kube_apiserver_patch)

# Create the config file for KinD
kind_config = """
# Refer https://github.com/kind-ci/examples/blob/master/gitlab/kind-config.yaml
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
networking:
  apiServerAddress: "0.0.0.0"

# Add an extra `kube-apiserver` argument to set a custom request timeout
kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
      extraArgs:
        request-timeout: {}s
        v: 10

# Add to the apiServer certSANs the name of the docker (dind) service in order
# to be able to reach the cluster through it
kubeadmConfigPatchesJSON6902:
  - group: kubeadm.k8s.io
    version: v1beta2
    kind: ClusterConfiguration
    patch: |
      - op: add
        path: /apiServer/certSANs/-
        value: host.docker.internal

nodes:
  - role: control-plane
    kubeadmConfigPatches:
      - |
        kind: JoinConfiguration
        patches:
          directory: /etc/kubernetes/manifests/patches
    extraMounts:
      - containerPath: /var/lib/kubelet/config.json
        hostPath: {}
      - containerPath: /etc/kubernetes/manifests/patches/kube-apiserver.yaml
        hostPath: {}
    extraPortMappings:
      # For postgres
      - containerPort: 32000
        hostPort: 5432
        listenAddress: 127.0.0.1
        protocol: TCP
  - role: worker
    kubeadmConfigPatches:
      - |
        kind: JoinConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "{}-{}/node-type=general"
    extraMounts:
      - containerPath: /var/lib/kubelet/config.json
        hostPath: {}
  - role: worker
    kubeadmConfigPatches:
      - |
        kind: JoinConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "{}-{}/node-type=general"
    extraMounts:
      - containerPath: /var/lib/kubelet/config.json
        hostPath: {}
""".format(
                apiserver_request_timeout_secs,
                ecr_auth_file.name,
                kube_apiserver_patch_file.name,
                platform_name,
                env_name,
                ecr_auth_file.name,
                platform_name,
                env_name,
                ecr_auth_file.name,
            )

[krish7919]

I changed the part

  - role: control-plane
    kubeadmConfigPatches:
      - |
        kind: JoinConfiguration
        patches:
          directory: /etc/kubernetes/manifests/patches

to

  - role: control-plane
    kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        patches:
          directory: /etc/kubernetes/manifests/patches

with no success.

[BenTheElder]

version: v1beta2

is that patch working? because if so I think you're using an older kubernetes version that doesn't have kubeadm config patches, the current version is v1beta3

I would recommend using a minimal config to narrow this down, probably also with verbosity increased while running kind.

I haven't had time to evaluate the kubeadm patch tooling wrt kind yet.