Incompatible with container-selinux > 2.188

[ameyp]

I'm running Fedora CoreOS 37.20221211.3.0, and I've installed both container-selinux-2.193.0 as well as k3s-selinux-1.2-2.el8 (from https://github.com/k3s-io/k3s-selinux/releases/download/v1.2.stable.2/k3s-selinux-1.2-2.el8.noarch.rpm). However, the k3s module doesn't appear to be getting loaded on the system, the output of semodule -l does not contain k3s. Additionally, if I try to install the module manually with semodule -i <path>, it fails due to a conflict:

[root@localhost core]# semodule -i /usr/share/selinux/packages/k3s.pp
Conflicting name type transition rules
Binary policy creation failed at /etc/selinux/targeted/tmp/modules/400/k3s/cil:135
Failed to generate binary
semodule:  Failed!

The error leads me to believe that one of rules specified in this package is conflicting with a rule in one of the existing modules.

[ameyp]

The actual problem that I'm trying to solve, is that /var/lib/rancher/k3s/storage is being created with the label container_var_lib_t instead of container_file_t, thereby causing the local-path provisioner to fail with permission denied.

[wranders]

Having this problem as well. From the CIL, the offending statement is

(typetransition container_runtime_t container_var_lib_t dir "snapshots" container_share_t)

which comes from https://github.com/k3s-io/k3s-selinux/blob/d97be9f6fcf2c467f7e826bc2c8c58da1375bfde/policy/centos8/k3s.te#L36

I tried looking through container-selinux for anything that would conflict from the previous integration of k3s rules, but didn't find anything.

This is occurring in 37.20221225.3.0 as well. Here are the package versions used:

• selinux-policy-37.16-1.fc37.noarch
• selinux-policy-targeted-37.16-1.fc37.noarch
• container-selinux-2:2.193.0-1.fc37.noarch

These were manually upgraded to the latest release version and made no difference.

[mysticaltech]

@brandond Having the exact same issue on OpenSUSE MicroOS latest!

Our project just came to an halt. See https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/issues/528#issuecomment-1401700309

When trying with semodule -vv -i /usr/share/selinux/packages/k3s.pp

It gives those issues:

Please, how do we fix this? It's kind of urgent as many daily installs are now failing.

[brandond]

The EL8 policy should work with container-selinux 2.167.0-1 and selinux-policy 3.14.3-67, which seems to work fine on both EL8 and EL9. We don't technically support CoreOS so I can't speak as to specific compatibility with any of the packages on that distro.

For MicroOS, it should work with container-selinux 2.164.2-1.1 and selinux-policy 20210716-3.1. If you're having trouble with MicroOS, please verify what version of those packages you're using. I suspect that something has been updated on the MicroOS side that is now conflicting, but I don't track that distro closely enough to know what off the top of my head.

[mysticaltech]

@brandond Yes, we suspected that too, as it was working yesterday, and since the latest release that just dropped, the problem showed up!

Will have a look at those packages' versions, and at ways to downgrade them.

[brandond]

With regards to selinux and local-path-provisioner, that is maintained by another team - the same team that owns Longhorn, to be specific. I believe they have an issue in their backlog to improve its compatibility with selinux and PSA.

[mysticaltech]

@brandond Indeed the versions on MicroOS are significantly more recent. I will investigate which of these packages were updated since yesterday.

[mysticaltech]

@brandond Thanks for your tips, as always very helpful!

After investigating the aforementioned packages here https://download.opensuse.org/tumbleweed/repo/oss/noarch/. We can see that it was container-linux that was upgraded just recently from 2.188 to 2.198.

So, downgrading with transactional-update --no-selfupdate -n pkg install --oldpackage https://download.opensuse.org/tumbleweed/repo/oss/noarch/container-selinux-2.188.0-2.1.noarch.rpm did work! 🎉

After that, applying semodule -vv -i /usr/share/selinux/packages/k3s.pp does successfully build the policy binary.

However, I blocked future automated upgrades of container-selinux with --no-selfupdate to avoid issues upon automatic upgrade back to 2.198 when the node is running.

But will that affect the security of the node and/or future versions of k3s? And is it really necessary to do so (lock the version)? Just so you understand the context, our nodes on Kube-Hetzner are meant to remain evergreen, by combining MicroOS's automatic upgrades, and the system-upgrade-controller for k3s.

Since the policy has been built successfully already, if we leave the automatic upgrade on for that package, would it fail to run under the more recent container-selinux 2.198 ?

[brandond]

The updates are listed at https://build.opensuse.org/request/show/1059620 but I don't see anything obvious that would break it, so I'll take some additional research. I believe that some of the suspicions I saw about the problem being the same as whatever's affecting Fedora CoreOS are correct.

[mysticaltech]

Ok, good to know! If you need any command outputs or logs, please don't hesitate! 🙏

[mysticaltech]

Sadly, it is confirmed; even the automatic upgrade breaks it because it tries to rebuild the policy! So locking of the version is important until the definite fix is shipped.

[galal-hussein]

I can reproduce the issue with the following versions:

• container-selinux-2.198.0-1.1.noarch
• selinux-policy-20221019-8.1.noarch
• k3s-selinux-1.2.stable.2-1.1.noarch

Compiling the policy revealed the same error, and the contexts are not applied to the processes or the files:

ls -ltrZ /var/lib/rancher/k3s/
total 4
drwxr-xr-x. 3 root root unconfined_u:object_r:var_lib_t:s0  106 Jan 25 19:42 data
drwxr-xr-x. 5 root root unconfined_u:object_r:var_lib_t:s0 4096 Jan 25 19:42 agent
drwx------. 8 root root system_u:object_r:var_lib_t:s0      148 Jan 25 19:42 server

Investigating the root cause now

[galal-hussein]

Please correct me if anything seems inaccurate or wrong, but I think I found the root cause, as @wranders mentioned, the problem was in that line:

Binary policy creation failed at /var/lib/selinux/targeted/tmp/modules/400/k3s/cil:135
...
(typetransition container_runtime_t container_var_lib_t dir "snapshots" container_share_t)

so looking further into what changed recently in container-selinux, I found this commit which seems to be the problem:

https://github.com/containers/container-selinux/commit/8ca4b89b82846d126000601f6b8530a962f0ecd3#diff-d87cfd6c86daf9204f212619e52210d06d7575d9b4ad27b41966ab3e1b80e2a8R536

specifically the conflict line will be here:

  filetrans_pattern($1, container_var_lib_t, container_file_t, dir, "snapshots")

which makes the snapshot dir writable, at first I was confused why this conflict didnt happen before in previous versions since the original line in container-selinux was:

 filetrans_pattern($1, container_var_lib_t, container_ro_file_t, dir, "snapshots")

and I found that container_share_t is basically an alias to container_ro_file_t:

typealias container_ro_file_t alias { container_share_t docker_share_t };

So I think the suggested fix will be changing the snapshots filetranspattern to match the container-selinux one.

cc @brandond @AkihiroSuda

[brandond]

Good find! I think we'll need to figure out how to handle this strategically, given that we do not need and cannot land this fix for EL8 or SLE Micro users (as those don't have the updated container-selinux version), but we don't have separate yum repos for EL8/Fedora Coreos and SLE Micro/MicroOS. We may need to create additional repos and add detection support to the install script.

[mysticaltech]

@brandond Not sure how it all works, but if the updated k3s-selinux RPM with the fix, could live on the rpm-testing.rancher.io repo for some time, it would be awesome!

[AkihiroSuda]

So I think the suggested fix will be changing the snapshots filetranspattern to match the container-selinux one.

Sorry what is the suggested fix? s/var_lib_t/container_file_t/ ? (Not container_ro_file_t, as the active snapshots have to be writable)

[mysticaltech]

FYI folks, the source PR https://github.com/containers/container-selinux/pull/82, where they explain their thinking.

[brandond]

if the updated k3s-selinux RPM with the fix, could live on the rpm-testing.rancher.io repo for some time, it would be awesome!

The fix would be appropriate to land there for MicroOS, but not for EL8.

[mysticaltech]

Just FYI @galal-hussein, I built it locally (from #37) for microos and installed on lastest MicroOS running container-selinux 2.198, and it worked!

However, I did notice this new relabeling activity going on, which was not there before to my knowledge. Please let me know if it's ok 🙏

And the final re-building of the policy (we do it after installing k3s, just in case):

[galal-hussein]

@AkihiroSuda yes the suggested fix will label s/var_lib_t/container_file_t/ instead of container_share_t https://github.com/k3s-io/k3s-selinux/pull/37/files#diff-a91aed3ea17f6544e61fe4e39d429a27316a95d80b7df73c3c43c76c9261a249R9

[galal-hussein]

Just FYI @galal-hussein, I built it locally (from #37) for microos and installed on lastest MicroOS running container-selinux 2.198, and it worked!

However, I did notice this new relabeling activity going on, which was not there before to my knowledge. Please let me know if it's ok pray

And the final re-building of the policy (we do it after installing k3s, just in case):

@mysticaltech good to know that its working, however I am not really sure about this relabeling activity, can you post the the final labels of these dirs for example:

• /var/lib/rancher/k3s/data
• /var/lib/rancher/k3s/agent/containerd

[mysticaltech]

@galal-hussein Here you go:

[galal-hussein]

@mysticaltech the labels looks fine as far as I can see

[mysticaltech]

@galal-hussein Perfect, thanks for confirming!

Maybe this relabeling happens during the installation of the RPM because we actually install k3s after (because of the need to reboot into the new MicroOS snapshot). So the k3s directories are not there yet.

And after having rebooted into the new snapshot, we install k3s without SELinux, and immediately build the policy again with semodule as shown in the second screenshot up the thread.

[mhjacks]

It appears we also have the same issue on CentOS 9 Stream which (at this time) has container-selinux-2.198.0-1.el9.noarch.

[mysticaltech]

@galal-hussein Very happy to see the 1.3.testing, just so we know what to expect, is this coming to stable anytime soon?

[rancher-max]

This is being tested currently and we'll update this issue with results as we find them! Hoping to be able to have it stable by March or April patch releases, pending results.

[johanneskastl]

Any news here? rke2-selinux has a fix in the works...

[balonik]

This bug affects "normal" Fedora as well.

  Running scriptlet: container-selinux-2:2.211.1-1.fc37.noarch 
  Upgrading        : container-selinux-2:2.211.1-1.fc37.noarch       
  Running scriptlet: container-selinux-2:2.211.1-1.fc37.noarch    
Conflicting name type transition rules
Binary policy creation failed at /var/lib/selinux/targeted/tmp/modules/200/k3s/cil:135
Failed to generate binary
/usr/sbin/semodule:  Failed!
  Cleanup          : container-selinux-2:2.209.0-1.fc37.noarch      
  Running scriptlet: container-selinux-2:2.209.0-1.fc37.noarch
  Verifying        : container-selinux-2:2.211.1-1.fc37.noarch      
  Verifying        : container-selinux-2:2.209.0-1.fc37.noarch

[eriklogtenberg]

Same on Fedora 38. These are the two packages the k3s installer wants to install:

• container-selinux-2.211.1-1.fc38.noarch.rpm
• k3s-selinux-1.2-2.el8.noarch.rpm

 Uitvoeren van scriptlet: k3s-selinux-1.2-2.el8.noarch                                                                                                    2/2 
 Installeren            : k3s-selinux-1.2-2.el8.noarch                                                                                                    2/2 
 Uitvoeren van scriptlet: k3s-selinux-1.2-2.el8.noarch                                                                                                    2/2 
Conflicting name type transition rules
Binary policy creation failed at /var/lib/selinux/targeted/tmp/modules/200/k3s/cil:135
Failed to generate binary
/usr/sbin/semodule:  Failed!