rancher-desktop nerdctl not working in windows 10 version under WSL2

nerdctl not working in windows 10 version under WSL2

Open hypeitnow opened this issue 1 year ago • 1 comments

Actual Behavior

❯  mkdir hello-world ; cd hello-world
❯ nano Dockerfile
❯ cat Dockerfile
FROM alpine
CMD ["echo", "Hello World!!"]
❯ nerdctl build --tag helloworld:v1.0 .
2022/08/20 14:16:47 Error parsing arguments: mkdir /sys/fs/cgroup/rancher-desktop: read-only file system
WARN[0000] lstat /root/Dockerfile: no such file or directory, using Containerfile as fallback 
FATA[0000] lstat /root/Containerfile: no such file or directory

Steps to Reproduce

mkdir hello-world ; cd hello-world
echo -e 'FROM alpine\nCMD ["echo", "Hello World!!"]'
nerdctl build --tag helloworld:v1.0 .

Result

No image is created.

Expected Behavior

Running nerdctl build from the tutorial https://docs.rancherdesktop.io/how-to-guides/hello-world-example fails, the expected bahaviour is to build Docker image.

Additional Information

The WSL distribution is currently using cgroups v1 since the kernelCommandLine = cgroup_no_v1=all option in .wslconfig file and mounting cgroups v2 mount -t cgroup2 cgroup2 /sys/fs/cgroup -o rw,nosuid,nodev,noexec,relatime,nsdelegate causes nerdctl to produce following errors:

2022-08-20T13:33:10.965Z: Registered distributions: Ubuntu-Preview,rancher-desktop,rancher-desktop-data
2022-08-20T13:33:11.109Z: Registered distributions: Ubuntu-Preview,rancher-desktop,rancher-desktop-data
2022-08-20T13:33:11.475Z: Registered distributions: Ubuntu-Preview,rancher-desktop,rancher-desktop-data
2022-08-20T13:33:11.652Z: Registered distributions: Ubuntu-Preview,rancher-desktop,rancher-desktop-data
2022-08-20T13:33:11.652Z: data distro already registered
2022-08-20T13:33:15.215Z: Did not find a valid mount, mounting /mnt/wsl/rancher-desktop/run/data
2022-08-20T13:33:21.626Z: Installing C:\Users\ok701f\AppData\Local\Programs\Rancher Desktop\resources\resources\linux\internal\rancher-desktop-guestagent as /mnt/c/Users/ok701f/AppData/Local/Programs/Rancher Desktop/resources/resources/linux/internal/rancher-desktop-guestagent into /usr/local/bin//rancher-desktop-guestagent ...
2022-08-20T13:33:21.636Z: Installing C:\Users\ok701f\AppData\Local\Programs\Rancher Desktop\resources\resources\linux\internal\trivy as /mnt/c/Users/ok701f/AppData/Local/Programs/Rancher Desktop/resources/resources/linux/internal/trivy into /usr/local/bin/trivy ...
2022-08-20T13:34:26.984Z: Waited more than 60 secs for nerdctl images to succeed. Giving up.

sudo mount --move /sys/fs/cgroup/unified /sys/fs/cgroup

❯ image ls
REPOSITORY    TAG    IMAGE ID    CREATED    PLATFORM    SIZE    BLOB SIZE
2022/08/20 15:41:52 Error removing mount directory /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.controllers: remove /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.controllers: operation not permitted
2022/08/20 15:41:52 Error removing mount directory /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.events: remove /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.events: operation not permitted
2022/08/20 15:41:52 Error removing mount directory /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.freeze: remove /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.freeze: operation not permitted
2022/08/20 15:41:52 Error removing mount directory /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.max.depth: remove /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.max.depth: operation not permitted
2022/08/20 15:41:52 Error removing mount directory /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.max.descendants: remove /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.max.descendants: operation not permitted
2022/08/20 15:41:52 Error removing mount directory /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.procs: remove /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.procs: operation not permitted
2022/08/20 15:41:52 Error removing mount directory /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.stat: remove /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.stat: operation not permitted
2022/08/20 15:41:52 Error removing mount directory /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.subtree_control: remove /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.subtree_control: operation not permitted
2022/08/20 15:41:52 Error removing mount directory /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.threads: remove /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.threads: operation not permitted
2022/08/20 15:41:52 Error removing mount directory /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.type: remove /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cgroup.type: operation not permitted
2022/08/20 15:41:52 Error removing mount directory /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cpu.stat: remove /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.3354090483/cpu.stat: 
operation not permitted

Rancher Desktop Version

1.5.1

Rancher Desktop K8s Version

1.2.4

Which container engine are you using?

containerd (nerdctl)

What operating system are you using?

Windows

Operating System / Build Version

Windows version 21H2 19044.1889

What CPU architecture are you using?

x64

Linux only: what package format did you use to install Rancher Desktop?

No response

Windows User Only

System Information

McAfee Agent
Version number: 5.7.4.399

McAfee Endpoint Security
Version number: 10.7 For module and content information, view the About page
in the McAfee Endpoint Security Client

Aug 20 '22 12:08 hypeitnow

Hello, thanks for the bug report! Can you help me understand your setup better? I'm confused about your use of cgroups v1 versus v2. Which one is default? Is there a reason that you modified which version you are using? It seems to me like the above description is for a problem that occurs after trying to work around the original problem, rather than describing the original problem. Is this true? Any help understanding this is much appreciated.

Aug 22 '22 21:08 adamkpickering

Thank you for your reply in the first place, it is not aimed at resolving any problems, I just wanted to switch to cgroups v2 from v1 since it's beneficial for the reasons described e.g here https://thenewstack.io/linux-cgroups-v2-brings-rootless-containers-superior-memory-management/ Just to make sure I switched wsl instance back to cgroups v1 but the issue I mentioned when trying to build containers stille persists WARN[0000] lstat /root/Dockerfile: no such file or directory, using Containerfile as fallback FATA[0000] lstat /root/Containerfile: no such file or directory

Aug 26 '22 08:08 hypeitnow

That's good to know. Is there anything else that is atypical about your environment? It's difficult to fix the problem when we can't reproduce it :)

Aug 26 '22 19:08 adamkpickering

I have the same issue. I've been trying to get K8s running with Docker, but only had partial success, so I decided to start over with a clean image and try nerdctl to see if that worked any better.

My setup is WSL2, Ubuntu image 22.04. Same OS, Rancher Desktop and K8s versions. I wanted to keep it clean, so I didn't mess with the configuration, just updated.

Given the issues I had previously, install seemed to work best when I had shutdown WSL, so I did that before running the installer. Docker Desktop for Windows was not running during install, or after.

ifconfig showed the following and I was able to ping all of the IPv4 addresses, which is a good sign:
johnl@DESKTOP-FDQ7CV5:~$ ifconfig
cni0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.42.0.1  netmask 255.255.255.0  broadcast 10.42.0.255
        inet6 fe80::24a4:48ff:fe03:8d74  prefixlen 64  scopeid 0x20<link>
        ether 26:a4:48:03:8d:74  txqueuelen 1000  (Ethernet)
        RX packets 7799  bytes 1590224 (1.5 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8997  bytes 2390098 (2.3 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.21.211.86  netmask 255.255.240.0  broadcast 172.21.223.255
        inet6 fe80::215:5dff:fe57:df78  prefixlen 64  scopeid 0x20<link>
        ether 00:15:5d:57:df:78  txqueuelen 1000  (Ethernet)
        RX packets 36032  bytes 48580939 (48.5 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5730  bytes 4888506 (4.8 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.42.0.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::8c21:34ff:fe02:bbf3  prefixlen 64  scopeid 0x20<link>
        ether 8e:21:34:02:bb:f3  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 5 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 24491  bytes 9588503 (9.5 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 24491  bytes 9588503 (9.5 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth19feb9db: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::f0c7:5ff:fec4:8363  prefixlen 64  scopeid 0x20<link>
        ether f2:c7:05:c4:83:63  txqueuelen 0  (Ethernet)
        RX packets 1331  bytes 126849 (126.8 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1432  bytes 151731 (151.7 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth861b7861: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::c8f:bff:fec0:6e9  prefixlen 64  scopeid 0x20<link>
        ether 0e:8f:0b:c0:06:e9  txqueuelen 0  (Ethernet)
        RX packets 70  bytes 6594 (6.5 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 141  bytes 17002 (17.0 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethb280045d: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::242b:cfff:fe97:e683  prefixlen 64  scopeid 0x20<link>
        ether 26:2b:cf:97:e6:83  txqueuelen 0  (Ethernet)
        RX packets 4297  bytes 1104056 (1.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5137  bytes 821957 (821.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethb8af9501: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::1489:1aff:fe2d:9364  prefixlen 64  scopeid 0x20<link>
        ether 16:89:1a:2d:93:64  txqueuelen 0  (Ethernet)
        RX packets 1637  bytes 112622 (112.6 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1667  bytes 114870 (114.8 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethf6daaa16: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::e4c1:4cff:fe22:b51b  prefixlen 64  scopeid 0x20<link>
        ether e6:c1:4c:22:b5:1b  txqueuelen 0  (Ethernet)
        RX packets 1223  bytes 106457 (106.4 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1756  bytes 215304 (215.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The installation instructions indicate that Helm, kubectl, nerdctl, Moby and Docker Compose will be available after install. When I ran kubectl version, it wasn't found and downloaded the latest version (I hope this doesn't cause issues). Here's the output when I checked on the other apps:

johnl@DESKTOP-FDQ7CV5:~/hello-world$ nerdctl build --tag helloworld:v1.0 .
WARN[0000] lstat /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.986227269/input.1118623100/Dockerfil
e: no such file or directory, using Containerfile as fallback
FATA[0000] lstat /sys/fs/cgroup/rancher-desktop/run/nerdctl-tmp.986227269/input.1118623100/Container
file: no such file or directory

johnl@DESKTOP-FDQ7CV5:~/hello-world$ helm version
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/johnl/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/johnl/.kube/config
version.BuildInfo{Version:"v3.9.1", GitCommit:"a7c043acb5ff905c261cfdc923a35776ba5e66e4", GitTreeState:"clean", GoVersion:"go1.17.5"}

johnl@DESKTOP-FDQ7CV5:~/hello-world$ moby version
Command 'moby' not found, did you mean:
  command 'goby' from deb goby-java (3.3.1+dfsg2-7)
Try: sudo apt install <deb name>

johnl@DESKTOP-FDQ7CV5:~/hello-world$ docker version
The command 'docker' could not be found in this WSL 2 distro.
We recommend to activate the WSL integration in Docker Desktop settings.
For details about using Docker Desktop with WSL 2, visit:
https://docs.docker.com/go/wsl2/

johnl@DESKTOP-FDQ7CV5:~/hello-world$ docker-compose version
The command 'docker-compose' could not be found in this WSL 2 distro.
We recommend to activate the WSL integration in Docker Desktop settings.
For details about using Docker Desktop with WSL 2, visit:
https://docs.docker.com/go/wsl2/

johnl@DESKTOP-FDQ7CV5:~$ containerd version
Command 'containerd' not found, but can be installed with:
sudo apt install containerd

Here are the results when I check for the apps in rancher-desktop:

~ # helm version
version.BuildInfo{Version:"v3.9.1", GitCommit:"a7c043acb5ff905c261cfdc923a35776ba5e66e4", GitTreeState:"clean", GoVersion:"go1.17.5"}

~ # nerdctl version
Client:
 Version:       v0.22.2
 OS/Arch:       linux/amd64
 Git commit:    2899222cb0715f1e5ffe356d10c3439ee8ee3ba4
 buildctl:
  Version:      v0.10.3
  GitCommit:    c8d25d9a103b70dc300a4fd55e7e576472284e31
FATA[0000] cannot access containerd socket "/run/containerd/containerd.sock": no such file or directory

~ # moby version
-sh: moby: not found

~ # docker version
Client:
 Version:           20.10.16
 API version:       1.41
 Go version:        go1.18.5
 Git commit:        aa7e414fdcb23a66e8fabbef0a560ef1769eace5
 Built:             Tue Aug  2 11:08:17 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

~ # docker-compose version
The command 'docker-compose' could not be found in this WSL 2 distro.
We recommend to activate the WSL integration in Docker Desktop settings.
For details about using Docker Desktop with WSL 2, visit:
https://docs.docker.com/go/wsl2/
~ # kubectl version
I0830 19:26:05.329229    7689 versioner.go:58] invalid configuration: no configuration has been provided
I0830 19:26:05.333519    7689 versioner.go:64] No local kubectl binary found, fetching latest stable release version
I0830 19:26:05.627855    7689 versioner.go:84] Right kubectl missing, downloading version 1.25.0
Downloading https://storage.googleapis.com/kubernetes-release/release/v1.25.0/bin/linux/amd64/kubectl
kubectl1.25.0 100% |████████████████████████████████████████| (45/45 MB, 29.381 MB/s) [1s:0s] done.
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
The connection to the server localhost:8080 was refused - did you specify the right host or port?

(NOTE: I didn't get the localhost error after kubectl was installed on Ubuntu; everything else was the same)

Do we need to copy, link, or manually install these apps? Please provide details.

Thanks, John

Aug 30 '22 19:08 johnl800

@sree792 thanks for your input! It is expected that kubectl will download on the first run - it is actually a symlink to kuberlr, which ensures that you have the correct version of kubectl for the version of the k8s API server you're running.

Not sure what the log about the connection to localhost:8080 being refused is about. Perhaps you didn't wait until everything was up? This is probably a red herring, and is not related to the topic of this issue anyways.

Aug 31 '22 16:08 adamkpickering

Thanks for responding.

Am I supposed to access K3s from my Ubuntu distro, the rancher-desktop distro, or both?

The install instructions aren't clear whether WSL2 and the distro's should be active prior to the install (but implies that WSL2 will be configured as a part of the set-up). I'm wondering if my issue is related to system configuration during installation.

re: connection to localhost: kubectl doesn't have that issue in the Ubuntu distro. In summary, the network is in better shape in Ubuntu; the RD distro appears to have more of the apps installed, but some have configuration issues and others are missing.

FYI, I also posted this on Slack, but I'm happy to communicate in either forum. Here's a link to the thread and an update from there: https://rancher-users.slack.com/archives/C0200L1N1MM/p1661916169255019?thread_ts=1661889117.612039&cid=C0200L1N1MM

Doug Merrill Are you running the rancher-desktop app during your testing? The links required are setup as the app initializes. If you are running the app, are you seeing any errors on the app?

johnl800 Yes, after install the rancher GUI shows that the Helm charts are complete and all services are active.

When rancher is down then the rancher-desktop distro in WSL2 is inactive, so I would not have been able to repeat the tests there. Notice that nerdctl and Docker are both available in the rancher distro, but not in Ubuntu. Also, it doesn't appear that containerd is available in either distro, which is likely, the cause of the issue.

I'm hoping for a solution from someone who knows how to solve the issue properly, instead of cobbing a solution myself.

Aug 31 '22 18:08 johnl800

Am I supposed to access K3s from my Ubuntu distro, the rancher-desktop distro, or both?

On Windows, the intended use of RD is from your own (i.e. not the rancher-desktop distro) WSL distro or from the regular Windows machine. We have things like rdctl because some workarounds require that the user goes into the rancher-desktop WSL distro, and some people are just curious and want to tweak things. You shouldn't be going into the rancher-desktop distro for your day-to-day work.

The install instructions aren't clear whether WSL2 and the distro's should be active prior to the install (but implies that WSL2 will be configured as a part of the set-up). I'm wondering if my issue is related to system configuration during installation.

I don't think that's likely, though maybe. RD should take whatever state the system is in at install time and either produce an error if the setup won't work, or make any needed changes if possible.

I'm hoping for a solution from someone who knows how to solve the issue properly, instead of cobbing a solution myself.

Unfortunately we haven't run into this before. And we have limited bandwidth, along with a need to deliver new features and fix other users' issues. If you're able to solve your issue, please post here so that others can benefit. Otherwise, we'll get to this as soon as we can :smile:

Sep 01 '22 22:09 adamkpickering

Hi Adam, I know that you're busy and I appreciate the time that you've spent to support me.

I'm new to WSL and K8s, so I'm still getting a handle on the details of the components, plumbing, etc. My intuition is that RD didn't install/link all of the components that were intended in the RD dist and/or my Ubuntu dist. I can install/link them manually, but I'm hoping that someone can provide the correct configuration (e.g. "containerd is installed in dist x:path y, with a sim-links in dist x:path z, dist a:path b"). Then I can correct the configuration without breaking the intended integration.

Can you provide that information for nerdctl and containerd?

Thanks, John

Sep 02 '22 00:09 johnl800

rancher-desktop rancher-desktop copied to clipboard

nerdctl not working in windows 10 version under WSL2

Actual Behavior

Steps to Reproduce

Result

Expected Behavior

Additional Information

Rancher Desktop Version

Rancher Desktop K8s Version

Which container engine are you using?

What operating system are you using?

Operating System / Build Version

What CPU architecture are you using?

Linux only: what package format did you use to install Rancher Desktop?

Windows User Only

rancher-desktop
rancher-desktop copied to clipboard