for-linux
for-linux copied to clipboard
docker
/sys/fs/cgroup/cpuset/docker/cpuset.cpus: no such file or directory due to noprefix mount option of CGROUP
Sorry, I have to re-report the issue same as https://github.com/moby/moby/issues/33594
Since I have not found the formal solution about it.
On the OS (Android , Chrome) which mount CPUSET with NO PREFIX option, we only have cpus, mems rather than cpuset.cpus or cpuset.mems.
I don't want change the OS behavior, but want make some fix on Docker. While I face some problems:
- the error log is
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:275: applying cgroup configuration for process caused "open /sys/fs/cgroup/cpuset/docker/cpuset.cpus: no such file or directory"": unknown.
container_linux.go:345 : I can find container_linux.go in the docker-ce project, but there is no line 345.. process_linux.go:275: I can not find such file at all!
So How did this error logs mean? How can I find the error spot ?
- I tried to modify the components/engine/pkg/sysinfo/sysinfo_linux.go and components/engine/vendor/github.com/containerd/cgroups/cpuset.go
I changed all “cpuset.xxx" to "xxx", but still failed. and I add logs in /vendor/github.com/containerd/cgroups/cpuset.go, seems server did not call the functions in the file before failure...
So How can I find the error spot, where should I modify
Thanks !
I have submitted a fix for runc, and now it can work in my case.
However, I found the similar codes in docker engine and containerd.
I could have fixed them as well. But I don't know how to trigger the issue and how to test. It seems that it can now on my case with only runc fix.
No idea when and how docker engine and containerd access the cpuset
@sophy228 hi :wave: -- yes I have a set of working patches for all runc, dockerd, containerd, and cadvisor (kubelet dependency)
I don't like how they're all variations of the same logic. Some of it is a bit scary. You can definitely tell that some code was copied between these projects.
Nice patch! Looks like @cyphar welcomes a fix. In order to trigger the other higher level bugs on android, you need to use containerd+dockerd with your patched runc. Building for arm64 / armv7 can be a little tricky. I got some help on the docker community #mobyproject slack channel.
I did plan to polish off all of these patches and submit them -- let me know if you want to collab :+1:
Hello. Same error after upgrading Fedora to version 31. Docker package: docker-ce-19.03.4-3.fc31.x86_64
@madrisan
sudo dnf install -y grubby && \
sudo grubby \
--update-kernel=ALL \
--args=”systemd.unified_cgroup_hierarchy=0"`
Background: https://medium.com/nttlabs/cgroup-v2-596d035be4d7
sudo dnf install -y grubby && \
sudo grubby \
--update-kernel=ALL \
--args="systemd.unified_cgroup_hierarchy=0"
++ Had to restart OS Worked!
All the ways did not work for me except this one:
- open /etc/default/grub as admin
- Append value of GRUB_CMDLINE_LINUX with
systemd.unified_cgroup_hierarchy=0 sudo -isudo grub2-mkconfig > /boot/efi/EFI/fedora/grub.cfgorsudo grub2-mkconfig > /boot/grub2/grub.cfgsudo reboot now
All the ways did not work for me except this one:
1. open /etc/default/grub as admin 2. Append value of **GRUB_CMDLINE_LINUX** with `systemd.unified_cgroup_hierarchy=0` 3. `sudo -i` 4. `sudo grub2-mkconfig > /boot/efi/EFI/fedora/grub.cfg` or `sudo grub2-mkconfig > /boot/grub2/grub.cfg` 5. `sudo reboot now`
Works!
Same problem on docker run hello-world.
OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused "open /sys/fs/cgroup/docker/cpuset.cpus.effective: no such file or directory"": unknown.
Client: Debug Mode: false
Server: Containers: 14 Running: 0 Paused: 0 Stopped: 14 Images: 3 Server Version: 19.03.5 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657 init version: fec3683 Security Options: seccomp Profile: default Kernel Version: 5.3.16-300.fc31.x86_64 Operating System: Fedora 31 (Thirty One) OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 14.62GiB Name: jupiter ID: C6CE:TU7L:AEMW:QF3D:2AUT:DS5I:YKZQ:WGYV:ZQXB:7GNB:QNBN:MTOU Docker Root Dir: /var/lib/docker Debug Mode: true File Descriptors: 24 Goroutines: 37 System Time: 2019-12-19T20:02:16.672999703+01:00 EventsListeners: 0 Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false
WARNING: No swap limit support WARNING: No kernel memory limit support WARNING: No kernel memory TCP limit support WARNING: No oom kill disable support WARNING: No cpu cfs quota support WARNING: No cpu cfs period support WARNING: No cpu shares support
All the ways did not work for me except this one:
1. open /etc/default/grub as admin 2. Append value of **GRUB_CMDLINE_LINUX** with `systemd.unified_cgroup_hierarchy=0` 3. `sudo -i` 4. `sudo grub2-mkconfig > /boot/efi/EFI/fedora/grub.cfg` or `sudo grub2-mkconfig > /boot/grub2/grub.cfg` 5. `sudo reboot now`
Omg ty! This one was hard to track down but this fixed mine too! :)
this save my day :) https://medium.com/@drpdishant/installing-docker-on-fedora-31-a073db823bb8
this save my day :) https://medium.com/@drpdishant/installing-docker-on-fedora-31-a073db823bb8
For me too! thanks.
Any side effects of this grub option?
systemd.unified_cgroup_hierarchy
When specified without an argument or with a true argument, enables the usage of
unified cgroup hierarchy (a.k.a. cgroups-v2). When specified with a false argument,
fall back to hybrid or full legacy cgroup hierarchy.
If this option is not specified, the default behaviour is determined during
compilation (the -Ddefault-hierarchy= meson option). If the kernel does
not support unified cgroup hierarchy, the legacy hierarchy will be used
even if this option is specified.```
I'm trying to fix this in chromeos under a chroot (crouton). It doesn't have grub so the unified cgroup hierarchy isn't really an option. I've tried symlinking cpus and mems to cpuset.xxx but the mount's options prevent this and I haven't been able to remount cgroups successfully. Are there any known workarounds? It seems like I'm so close.
systemd.unified_cgroup_hierarchy When specified without an argument or with a true argument, enables the usage of unified cgroup hierarchy (a.k.a. cgroups-v2). When specified with a false argument, fall back to hybrid or full legacy cgroup hierarchy. If this option is not specified, the default behaviour is determined during compilation (the -Ddefault-hierarchy= meson option). If the kernel does not support unified cgroup hierarchy, the legacy hierarchy will be used even if this option is specified.```
Are you planning to support the unified_hierarchy some day?
@jifalops I was working on this from an android perspective. My understanding is the ChromeOS issue is similar.
I had to patch runc, containerd, dockerd, and the kubelet's cadvisor in order to support noprefix crgroup mounts.
I got everything working but haven't had the time to publish and push through the patches yet. You may have luck patching your own builds.
Hooray for systemd breaking things without providing adequate fallback/legacy API.
systemd supports both the versions of cgroup, so there's no break at all. It's up to the distribution maintainers to decide which version to choose by default. And as pointed out in this same thread, switching from the one version to the other one is just a question of adding a kernel parameter at boot.
Yeah, it's like swapping a nuclear reactor engine. You have to shut it down first. Easy as it gets. That's no compatibility, that's called an offline swap.
The compatibility (or a switch to the unified cgroup hierarchy) have to be implemented in Docker and is in progress. So let's give them the time they need of just switch back to the old implementation at boot time, if you use the only (afaiks) Linux distribution that has chosen to break Docker in order to push their alternative container engine.
Systemd uses cgroups, the newer cgroups V2 is the one not backwards compatible or fully functional alongside V1, its around since 2016. If you configure systemd to setup V2 then you have to live with consequences (apps incompatible with V2), if not you can pick V1 with a boot parameter.
So Linux has multiple Options, Systemd supports all of those, Docker doesnt (yet), you are overwhelmed with that. Blame Systemd all you want, but maybe do that in the right cult.
Here we go with SJW (systemd justice warriors) defending dubious systemd choices. Color me surprised.
Now there is an official article from Fedora Magazine: Docker and Fedora 32 instructs how to solve the error
Hi guys! I just installed Docker 20.10 beta1, while it had some issues configuring the bridge, adding it manually (brctl addbr docker1) and setting it in daemon.json: {"bridge": "docker1"}, docker works with cgroups v2, at least with the hello-world image :D
@jifalops I was working on this from an android perspective. My understanding is the ChromeOS issue is similar.
I had to patch runc, containerd, dockerd, and the kubelet's cadvisor in order to support noprefix crgroup mounts.
I got everything working but haven't had the time to publish and push through the patches yet. You may have luck patching your own builds.
Is there any info on how to fix this? I'm trying to get this working for ChromeOS & Chromebrew: https://github.com/chromebrew/chromebrew/pull/7828
