upgrades are done regardless of running version or --disable-upgrade-check set

[itsek]

k0sctl upgrades cluster even if it runs the same version as the desired version, steps to reproduce:

Install Cluster using some older Version (1.27.0), do modify k0sctl config file to set desired Version to 1.27.2, run k0sctl apply --config <path_to_config_file> Now Cluster is on 1.27.2 - so far so good

now run k0sctl apply --config <path_to_config_file> again, with the same k0s configfile set to 1.27.2 as the targetversion, k0sctl will upgrade 1.27.2 over 1.27.2 ?!

k0sctl apply --config <path_to_config_file> --disable-upgrade-check --> will also do the upgrade

Console out:

(...)
INFO [ssh] <###ip###>:22: is running k0s controller version 1.27.2+k0s.0
WARN [ssh] <###ip###>:22: k0s will be upgraded
(...)

Config file:

(...)
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
    name: <### name-redacted ###>
spec:
    hosts:
        - ssh:
            address: <### ip-redacted ###>
            user: ${USER}
            port: 22
            keyPath: ~/.ssh/id_ed25519
          role: controller
          privateAddress: <### ip-redacted ###>
          uploadBinary: true
          k0sBinaryPath: <### path redacted ###>/k0s-v1.27.2+k0s.0-amd64
          installFlags:
            - --kubelet-extra-args='--config=<### path redacted ###>.yaml'
(...)
        - ssh:
            address: <### ip-redacted ###>
            user: ${USER}
            port: 22
            keyPath: ~/.ssh/id_ed25519
          role: worker
          privateAddress: <### ip-redacted ###>
          uploadBinary: true
          k0sBinaryPath: <### path redacted ###>/k0s-v1.27.2+k0s.0-amd64
          installFlags:
            - --kubelet-extra-args='--config=<### path redacted ###>.yaml'

    k0s:
        version: v1.27.2+k0s.0
        dynamicConfig: false
        config:
            apiVersion: k0s.k0sproject.io/v1beta1
            kind: ClusterConfig
            metadata:
                name: <### name ###>
            spec:
                api:
                    externalAddress: <### ip ###>
(...)

[kke]

This is because you have given k0sbinarypath. K0sctl does not know what version that binary is.

There is supposed to be a check to see if the file has changed:

https://github.com/k0sproject/k0sctl/blob/04a76d8461934b92ed77d89bebb2cc90a5afce89/phase/gather_k0s_facts.go#L197-L200

Maybe the change detection isn't working? There should be more info in the debug log and a reason why it decided to run an upgrade (which is essentially just k0s stop => switch binary => k0s start):

https://github.com/k0sproject/k0sctl/blob/main/pkg/apis/k0sctl.k0sproject.io/v1beta1/cluster/host.go#L579-L587

[itsek]

Thanks for the Hints

problem was that the client executing k0sctl and the cluster-nodes differed in actual time and the timezone, I'll investige the exact error...

Now it leaves the question why the --disable-upgrade-check flag got ignored in the first place?

[itsek]

Not sure what happens, but here are the timestamp of the binary on the mashine executing k0sctl:

stat k0s-v1.27.2+k0s.0-amd64
  File: k0s-v1.27.2+k0s.0-amd64
  Size: 230559893       Blocks: 450320     IO Block: 4096   regular file
Device: fe01h/65025d    Inode: 2258970     Links: 1
Access: (0775/-rwxrwxr-x)  Uid: (1101324/adminchs)   Gid: (1100000/adminusers)
Access: 2023-06-27 11:38:21.708117374 +0200
Modify: 2023-06-27 11:37:50.362945308 +0200
Change: 2023-06-27 11:37:50.362945308 +0200
 Birth: 2023-06-25 09:31:43.851764026 +0200

And on one of the Cluster Nodes:

sudo stat /usr/local/bin/k0s
  File: /usr/local/bin/k0s
  Size: 230559893       Blocks: 450320     IO Block: 4096   regular file
Device: fe05h/65029d    Inode: 532032      Links: 1
Access: (0700/-rwx------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-06-27 11:48:11.728416656 +0200
Modify: 2023-06-27 11:48:11.596482657 +0200
Change: 2023-06-27 11:48:11.716422656 +0200
 Birth: 2023-02-08 12:29:57.312792852 +0100

Now the debug output of k0sctl apply:

(...)
DEBU[0016] [ssh] 10.241.4.52:22: file modtimes for /data/resources/kubernetes/k0sctl_binaries/k0s-v1.27.2+k0s.0-amd64 differ (2023-06-27 11:37:50.362945308 +0200 CEST vs 2023-06-27 11:47:47.779017276 +0200 CEST)
DEBU[0016] [ssh] 10.241.4.52:22: marked for upgrade because of a static k0s binary path /data/resources/kubernetes/k0sctl_binaries/k0s-v1.27.2+k0s.0-amd64
(...)

Not sure where the second modtime (after vs) reported by k0sctl come from

[kke]

It's possible the sudo output parsing / formatting has some bug in it or it fails to consider timezones.

Stat is run with these params (depending on if --format is found in --help)

    file_info=$(stat --format="0%a %s %Y" "$path" 2> /dev/null)
    file_info=$(stat -f "%Mp%Lp %z %m" "$path" 2> /dev/null)

The --disable-upgrade-check is for disabling checking for new k0sctl version.

[kke]

Or maybe the touch failed that should set the modtime after upload 🤔

[itsek]

I'll assume that touch fails for some reason

As a quick fix I'll run this ansible playbook after upgrade, which simply gets the last modified timestamp from my local file and applies it to the remote binary

---
- hosts: all
  vars:
    local_source_file: <my_mashine_running_k0sctl>/k0s-v1.27.3+k0s.0-amd64
    remote_target_file: /usr/local/bin/k0s

  tasks:
    - name: get timestamp of local file
      delegate_to: localhost
      register: reference_timestamp
      ansible.builtin.shell: stat -c "%y" {{ local_source_file }}

    - name: appyl timestamp to remote target file
      become: true
      ansible.builtin.shell: touch -d "{{ reference_timestamp.stdout }}" "{{ remote_target_file }}"

[kke]

Cool if it works. I'll investigate the touch!