Feature: Allow advanced configuration of k3s from the UI

[stellirin]

We have a bare metal k3s cluster and we would like to replicate its configuration in Rancher Desktop. One example of what we would like to replicate is the use of --disable=traefik so that we can install Traefik ourselves. Reasons:

• install v2 on versions of k3s where v1 is still the default
• additional listeners to export a PostgreSQL service on port 5432

Not every configuration option makes sense to be configurable, but in this specific case it would map well to a checkbox in the UI (same for the other components that can be disabled with this flag).

Alternatives:

1. a simple text box that we can write the k3s flags and their options directly, accepting the risk that typos could cause k3s startup to fail
2. similar to above but to influence the Traefik installation directly (helm options etc.)

Check boxes to disable components and a second advanced text box for k3s flags seems like the best options for flexibility, and can be implemented separately.

[JockDaRock]

I am totally for this as working with Istio on rancher-desktop is difficult if you can't disable traefik. would be also interesting if you can disable the embedded (klipper?) lb as well :)

[kt81]

It's nice to have the ability to configure any options. (alt. 1) I'd like to give k3s the option of --service-node-port-range but other people may not.

[stellirin]

note for me:

• #951 is taged for v1.0.0
• #926 is a workaround and was released in v0.7.0

On MacOS we can now manually edit ~/Library/Application\ Support/rancher-desktop/lima/0/lima.yaml to add any k3s server flags such as:

env:
  K3S_EXEC: --disable=traefik

[cvstebut]

Does anybody know how to set the variable on Windows / WSL? Thanks!

[jandubois]

On MacOS we can now manually edit ~/Library/Application\ Support/rancher-desktop/lima/0/lima.yaml

@stellirin Starting with RD 0.7.0 you should put any such customization into ~/Library/Application\ Support/rancher-desktop/lima/_config/override.yaml instead. That file will not be written to by RD itself, so customizations are protected from any changes RD makes to lima.yaml.

@cvstebut It is possible to make the same change with the WSL2 distro, but I don't know off-hand the best way to configure it. @mook-as should be able to provide an answer next week.

[mook-as]

There is no supported way of doing this on Windows yet. As a workaround, you can add K3S_EXEC=… to /etc/rc.conf (see the section around line 108, Service Configuration Variables); however, that will be clobbered when we update the base distribution (roughly, on Rancher Desktop updates).

Filed #1142 to properly support this.

[cvstebut]

Thanks a lot! For my use case this works great. Permanence is nice, but now I know where to configure it when needed. I have just successfully tested it:

• opened the WSL distro "rancher-desktop" using the windows terminal
• added a line to /etc/rc.conf K3S_EXEC="--disable=traefik --disable=servicelb"
• "Reset Kubernetes" in the Rancher Desktop UI

Works as desired :-)

[zerus83]

On MacOS we can now manually edit ~/Library/Application\ Support/rancher-desktop/lima/0/lima.yaml

@stellirin Starting with RD 0.7.0 you should put any such customization into ~/Library/Application\ Support/rancher-desktop/lima/_config/override.yaml instead. That file will not be written to by RD itself, so customizations are protected from any changes RD makes to lima.yaml.

@cvstebut It is possible to make the same change with the WSL2 distro, but I don't know off-hand the best way to configure it. @mook-as should be able to provide an answer next week.

Hi i tried using the override yaml method but keep getting errors.

override.yaml: env: K3S_EXEC: "--disable=traefik -flannel-backend=none"

Error encountered:

/lib/rc/sh/openrc-run.sh: /etc/environment: line 7: -flannel-backend=none: not found

• k3s: error loading /etc/init.d/k3s
• ERROR: k3s failed to start

any idea how to resolve this? thanks!

[cvstebut]

@zerus83 Could be just a spelling mistake: Try "--flannel-backend..." instead of just "-flannel-backend ..."

[zerus83]

@zerus83 Could be just a spelling mistake: Try "--flannel-backend..." instead of just "-flannel-backend ..."

apologies , forgot to elaborate- actually i try a few combinations like the following:

K3S_EXEC: --cluster-cidr=10.66.0.0/16 --disable=traefik K3S_EXEC: "--cluster-cidr=10.66.0.0/16 --disable=traefik" K3S_EXEC: "--cluster-cidr=10.66.0.0/16 -disable=traefik"

whenever i added more than 1 option in the arguments, the 2nd option will have the error.

if i were to switch the order to K3S_EXEC: --disable=traefik --cluster-cidr=10.66.0.0/16, i will get the below error:

lib/rc/sh/openrc-run.sh: /etc/environment: line 7: --cluster-cidr=10.66.0.0/16: not found

i am using mac M1 silicon.

[jandubois]

Error encountered:

/lib/rc/sh/openrc-run.sh: /etc/environment: line 7: -flannel-backend=none: not found

This is a bug in /etc/init.d/k3s: it executes source /etc/environment, which is invalid because the file is not using shell syntax.

[micahmo]

@kt81

I'd like to give k3s the option of --service-node-port-range but other people may not.

+1, I have this use case as well. Note that you can use K3S_EXEC for this purpose.

K3S_EXEC="--kube-apiserver-arg service-node-port-range=1-65535"

[jandubois]

The option to disable traefik is included in the 1.1.0 release. A more general mechanism is still outstanding.

[marcellodesales]

🔧 Config

• The settings described for the overrides.yaml worked flawlessly!
  • https://github.com/rancher-sandbox/rancher-desktop/issues/578#issuecomment-997454430
• Here's how I enabled feature gate CronJobTimeZone!
  • https://kubernetes.io/docs/reference/command-line-tools-reference/_print/#feature-gates
  • https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/

# https://github.com/rancher-sandbox/rancher-desktop/issues/578#issuecomment-996557436
env:
  K3S_EXEC: --kube-apiserver-arg --feature-gates=CronJobTimeZone=true

🔊 K3S Logs

• Just make sure nothing went wrong by looking at the logs

tail -f ~/Library/Logs/rancher-desktop/k3s.log

• I confirmed by verifying that the feature gates was seen on the logs: --feature-gates=CronJobTimeZone=true

time="2022-08-12T19:15:57Z" level=info msg="Acquiring lock file /var/lib/rancher/k3s/data/.lock"
time="2022-08-12T19:15:57Z" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/230cb7b95d48672d3d564dfde9c7b6230bccc82b2a3298f565dc3cca2a06b2eb"
time="2022-08-12T19:15:59Z" level=info msg="Found ip 192.168.86.249 from iface rd0"
time="2022-08-12T19:15:59Z" level=info msg="Starting k3s v1.24.3+k3s1 (990ba0e8)"
time="2022-08-12T19:15:59Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
time="2022-08-12T19:15:59Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
time="2022-08-12T19:15:59Z" level=info msg="Database tables and indexes are up to date"
time="2022-08-12T19:15:59Z" level=info msg="Kine available at unix://kine.sock"
time="2022-08-12T19:15:59Z" level=info msg="Reconciling bootstrap data between datastore and disk"
time="2022-08-12T19:15:59Z" level=info msg="Running kube-apiserver --advertise-address=192.168.86.249 --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --etcd-servers=unix://kine.sock --feature-gates=CronJobTimeZone=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
time="2022-08-12T19:15:59Z" level=info msg="Tunnel server egress proxy mode: agent"

🐛 Bug setting CronJob timezones

As the feature gate is still in Alpha and it got enabled properly, given the validations, it fails to set the timezone for CronJobs.

• still need to setup the timezone for the kube-controller-manager... :(
  • https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
  • Names starting with Etc/* as described at https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#time-zones

The CronJob "timecard-cron-job-local-localhost-prdt-ppd-loc" is invalid: spec.timeZone: Invalid value: "Etc/GMT+8": unknown time zone Etc/GMT+8

• Even the default fails

The CronJob "timecard-cron-job-local-localhost-prdt-ppd-loc" is invalid: spec.timeZone: Invalid value: "Etc/UTC": unknown time zone Etc/UTC

❓ What's the current support for this feature?

@gaktive

• Is there any way to set a value? or is it just broken?
• Is there any workaround?

[arslanakhtar61]

Would be nice to have a disabled checkbox in the rancher-desktop UI for KlipperLB, the same as with Traefik

[stahir80td]

I am trying to pass in multiple arguments to kubeapi-server. I edited /etc/rc.conf, I am seeing the values being passed in to the logs but not as I want them.

Goal: I need to pass in two parameters, not one.

Option 1: K3S_EXEC="--kube-apiserver-arg oidc-client-id=value1 oidc-issuer-url=value2"

Outcome: Only adds first parameter (oidc-client-id) to kubeapi-server param

Option 2: K3S_EXEC="--kube-apiserver-arg oidc-client-id=value1" K3S_EXEC="--kube-apiserver-arg oidc-issuer-url=value2"

Outcome: Only adds 2nd parameter (oidc-issuer-url) to kubeapi-server params

Any suggestions on how to pass in multiple parameters to kubeapi-server using /etc/rc.conf Many Thanks,

[AxelTahmid]

🐛 Bug setting CronJob timezones

still facing the unknown time zone issue, running 1.28.4

[rbofa]

We are not able to replicate rancher desktop with metallb. Traefik disabled on latest rancher desktop ui, but still servicelb now letting Loadbalancer type service to execute on linux rancher desktop ui.

Tried on rancher desktop cluster information UI screen, by setting the annotation k3s argument as ["--disable", "traefik","--disable", "servicelb" ] restrted the desktop. Still exposed service type: LoadBalancer is not working/reaching. Can see svclb- pods for the same service.

kind: Service apiVersion: v1 metadata: name: foo-service-2 spec: type: LoadBalancer selector: app: http-echo ports: port: 5678