Calico node check includes false negatives for modern Linux distros

[ahrkrak]

Expected Behavior

Expect Calico node check to not return an error if the OS supports all the features it needs.

Current Behavior

calicoctl node checksystem fails ipt_set, xt_icmp, xt_icmp6, and xt_rpfilter checks.

# calicoctl node checksystem  
Checking kernel version...
5.4.83-flatcar      OK
Checking kernel modules...
ip_set              OK
ip_tables           OK
ip6_tables          OK
ipt_REJECT          OK
ipt_rpfilter        OK
WARNING: Unable to detect the ipt_set module as Loaded/Builtin module or lsmod
ipt_set             FAIL
nf_conntrack_netlink OK
xt_addrtype         OK
xt_conntrack        OK
WARNING: Unable to detect the xt_icmp module as Loaded/Builtin module or lsmod
xt_icmp             FAIL
WARNING: Unable to detect the xt_icmp6 module as Loaded/Builtin module or lsmod
xt_icmp6            FAIL
xt_ipvs             OK
xt_mark             OK
xt_multiport        OK
WARNING: Unable to detect the xt_rpfilter module as Loaded/Builtin module or lsmod
xt_rpfilter         FAIL
xt_set              OK
xt_u32              OK
System doesn't meet one or more minimum systems requirements to run Calico

Possible Solution

Update node check logic. If the "ipt_set" module is built-in (as is the case with Flatcar and likely these other distros too), then "sudo modprobe ipt_set" is a no-op and it does not show up in "lsmod | grep ipt_set" We believe the checks for "xt_icmp", "xt_icmp6", and "xt_rpfilter" are outdated as those modules don't exist / aren't available in Flatcar, Debian or Fedora. Alternate methods of testing for the presence of required features should be used, or the tests removed, or at a minimum the summary text changed to indicate that the results do not necessarily mean minimum system requirements are not met.

Steps to Reproduce (for bugs)

1. install Calico on latest Flatcar, Ubuntu, Fedora, or Debian
2. run calicoctl noode checksystem
3. observe output

Context

Impacted a Flatcar user that was trying to deploy Calico. Has also been reported by kubespray users - see https://github.com/kubernetes-sigs/kubespray/issues/6289

Your Environment

• Calico version: 3.16
• Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes
• Operating System and version: Flatcar 2605 (also present on Ubuntu 18.04 and others)

[caseydavenport]

Alternate methods of testing for the presence of required features should be used, or the tests removed, or at a minimum the summary text changed to indicate that the results do not necessarily mean minimum system requirements are not met.

Yep, I agree with all of this.

[caseydavenport]

@frozenprocess I've seen this one impacting a number of folks recently, if you're interested in giving it a go!

[frozenprocess]

For ipt_set I think it would be ok to PASS the test if xt_SET is present.

core@localhost ~ $ modinfo xt_set -F alias       
ip6t_SET
ipt_SET
ip6t_set
ipt_set
xt_SET
core@localhost ~ $ modinfo xt_set -F description
Xtables: IP set match and target module

I think it is safe to pass xt_icmp and xt_icmp6 if icmp is present in ip_tables_matches file.

sudo cat /proc/net/ip_tables_matches | grep icmp
icmp

WDYT?

[DanSibbernsen]

Hi all, FWIW I'm also experiencing this same problem on Suse Enterprise Linux 15.2 (SLES 15.2). Thanks for digging into this, helps to know I can ignore this problem when trying to stand up canal.

[caseydavenport]

I believe @frozenprocess made some good improvements to this for v3.19. I'm going to close this for now, but please report back if this is still an issue once we release v3.19.

[roooxvee]

Hello.

Same problem on Rocky9 (RHEL9).

[root@k8s-test ~]$ calicoctl version Client Version: v3.22.0 Git commit: a86e41d02 Cluster Version: v3.22.0 Cluster Type: k8s,bgp,kdd,kubeadm

[root@k8s-test ~]$ calicoctl node checksystem Checking kernel version... 5.14.0-70.13.1.el9_0.x86_64 OK Checking kernel modules... xt_rpfilter OK nf_conntrack_netlink OK xt_addrtype OK xt_icmp6 OK ip_set OK ip_tables OK ipt_rpfilter OK xt_set OK vfio-pci OK ipt_REJECT OK WARNING: Unable to detect the ipt_set module as Loaded/Builtin module or lsmod ipt_set FAIL xt_icmp OK WARNING: Unable to detect the xt_u32 module as Loaded/Builtin module or lsmod xt_u32 FAIL ipt_ipvs OK xt_bpf OK xt_conntrack OK ip6_tables OK xt_mark OK xt_multiport OK

Also I have xt_set loaded, and ipt_set is still failing.... [root@k8s-test ~]$ lsmod | grep xt_set xt_set 16384 4 ip_set 61440 3 ip_set_hash_ip,xt_set,ip_set_hash_net

Any updates?