k0sctl icon indicating copy to clipboard operation
k0sctl copied to clipboard
verified publisher icon k0sproject

Getting `spec.k0s.config fails validation` without any feedback

Open muvaf opened this issue 1 year ago • 2 comments

Before creating an issue, make sure you've checked the following:

  • [X] You are running the latest released version of k0s
  • [X] Make sure you've searched for existing issues, both open and closed
  • [X] Make sure you've searched for PRs too, a fix might've been merged already
  • [X] You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

Linux 5.15.0-105-generic k0sproject/k0s#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Version

v1.29.4+k0s.0

Sysinfo

`k0s sysinfo` 
Machine ID: "c7c2bff592705c19357992184ea068ebe039a15db6636739cedf62fd346a0788" (from machine) (pass)
Total memory: 62.6 GiB (pass)
Disk space available for /var/lib/k0s: 438.5 GiB (pass)
Name resolution: localhost: [127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 5.15.0-105-generic (pass)
  Max. file descriptors per process: current: 1048576 / max: 1048576 (pass)
  AppArmor: active (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass)
    CONFIG_CPUSETS: Cpuset support: built-in (pass)
    CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass)
    CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: module (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: module (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: module (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
      CONFIG_LLC: module (pass)
      CONFIG_STP: module (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

The first run of k0sctl apply --config k0sctl.yaml succeeds and cluster comes up. When I run k0sctl apply --config k0sctl.yaml for the second time for the same config file, I get the following error:

INFO ==> Apply failed
FATA apply failed - log file saved to /Users/muvaf/Library/Caches/k0sctl/k0sctl.log: spec.k0s.config fails validation:

No details on what the error was and log file also ends with the same line.

When I SSH into my machine and create a YAML only with fields under spec.k0s.config, k0s config validate command exits with success.

Here is the redacted version of my config.yaml:

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: eu-1
spec:
  hosts:
    - ssh:
        address: <domain pointing to machine ip>
        user: root
        port: 22
        keyPath: /Users/muvaf/.ssh/id_ed25519
      role: controller+worker
      noTaints: true
  k0s:
    version: "v1.29.4+k0s.0"
    versionChannel: stable
    dynamicConfig: false
    config:
      spec:
        api:
          address: <machine ip>
          externalAddress: <domain pointing to machine ip>
        storage:
          type: etcd
        network:
          provider: calico
          # provider: custom
          podCIDR: 10.244.0.0/16
          serviceCIDR: 10.96.0.0/12
        telemetry:
          enabled: false

Steps to reproduce

  1. Get a machine and run k0sctl apply --config k0sctl.yaml with YAML I provided above, after IP and domain changes.
  2. After it succeeds, run the command again.
  3. Get the error above.

Expected behavior

Command to succeed.

Actual behavior

Fails with spec.k0s.config fails validation

Screenshots and logs

Logs redacted:

...
time="29 Apr 24 16:24 +03" level=debug msg="Preparing phase 'Validate facts'"
time="29 Apr 24 16:24 +03" level=info msg="\x1b[32m==> Running phase: Validate facts\x1b[0m"
time="29 Apr 24 16:24 +03" level=debug msg="Preparing phase 'Download k0s binaries to local host'"
time="29 Apr 24 16:24 +03" level=debug msg="Preparing phase 'Upload k0s binaries to hosts'"
time="29 Apr 24 16:24 +03" level=debug msg="Preparing phase 'Download k0s on hosts'"
time="29 Apr 24 16:24 +03" level=debug msg="[ssh] redacted.net:22: k0s version on target host is already v1.29.4+k0s.0"
time="29 Apr 24 16:24 +03" level=debug msg="Preparing phase 'Upload files to hosts'"
time="29 Apr 24 16:24 +03" level=debug msg="Preparing phase 'Install k0s binaries on hosts'"
time="29 Apr 24 16:24 +03" level=debug msg="Preparing phase 'Prepare ARM nodes'"
time="29 Apr 24 16:24 +03" level=debug msg="Preparing phase 'Configure k0s'"
time="29 Apr 24 16:24 +03" level=debug msg="using provided k0s config"
time="29 Apr 24 16:24 +03" level=debug msg="[ssh] redacted.net:22: executing `mktemp`"
time="29 Apr 24 16:24 +03" level=debug msg="[ssh] redacted.net:22: /tmp/tmp.6OznpAzyZP"
time="29 Apr 24 16:24 +03" level=debug msg="[ssh] redacted.net:22: executing `mktemp 2> /dev/null`"
time="29 Apr 24 16:24 +03" level=debug msg="[ssh] redacted.net:22: /tmp/tmp.1gLxv8Uo0p"
time="29 Apr 24 16:24 +03" level=debug msg="[ssh] redacted.net:22: executing `cat > /tmp/tmp.1gLxv8Uo0p`"
time="29 Apr 24 16:24 +03" level=debug msg="[ssh] redacted.net:22: writing 431 bytes to command stdin"
time="29 Apr 24 16:24 +03" level=debug msg="[ssh] redacted.net:22: executing `install -D -m 0600 -- /tmp/tmp.1gLxv8Uo0p /tmp/tmp.6OznpAzyZP`"
time="29 Apr 24 16:24 +03" level=info msg="[ssh] redacted.net:22: validating configuration"
time="29 Apr 24 16:24 +03" level=debug msg="[ssh] redacted.net:22: comparing k0s version v1.29.4+k0s.0 with >= v1.23.1+k0s.0"
time="29 Apr 24 16:24 +03" level=debug msg="[ssh] redacted.net:22: comparison result true"
time="29 Apr 24 16:24 +03" level=debug msg="[ssh] redacted.net:22: executing `/usr/local/bin/k0s config validate --config \"/tmp/tmp.6OznpAzyZP\"`"
time="29 Apr 24 16:24 +03" level=debug msg="[ssh] redacted.net:22: executing `rm -f -- /tmp/tmp.6OznpAzyZP 2> /dev/null`"
time="29 Apr 24 16:24 +03" level=info msg="\x1b[31m==> Apply failed\x1b[0m"
time="29 Apr 24 16:24 +03" level=fatal msg="apply failed - log file saved to /Users/muvaf/Library/Caches/k0sctl/k0sctl.log: spec.k0s.config fails validation:"

Additional context

The target machine is a bare metal server.

muvaf avatar Apr 28 '24 14:04 muvaf

Can you share the k0sctl log file so we can see what k0sctl was doing before it failed? Sounds related to #679.

twz123 avatar Apr 29 '24 07:04 twz123

@twz123 Sure, added to the logs section in the description.

muvaf avatar Apr 29 '24 13:04 muvaf

#692 and #709 may shed some light on the validation problems in the future.

kke avatar May 14 '24 07:05 kke