Add support for running Sysbox on Windows (via WSL2)

[doggy8088]

I'm using Docker for Windows with WSL 2 integration. Can I use sysbox? Do you have any guidance about this environment?

[rodnymolina]

@doggy8088 thanks for giving Sysbox a shot.

As you may have noticed from our documentation, at the moment we only support native linux deployments. WSL2 & Ubuntu is on our roadmap but we haven't spent any cycles on it yet. If you decide to give it a try, please let us know how it works for you, that would be really helpful.

[felipecrs]

I'm not able to install sysbox, since the WSL kernel is older than the required.

[felipecrs]

This might help: https://github.com/microsoft/WSL2-Linux-Kernel/issues/82

[bittelc]

@braedongough plz review

[felipecrs]

News in this one:

Microsoft released the WSL2 Linux 5.4 Kernel, but the sysbox installation still doesn't succeeds:

❯ uname -r
5.4.72-microsoft-standard-WSL2
❯ sudo apt install ./sysbox_0.2.1-0.ubuntu-focal_amd64.deb
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'sysbox' instead of './sysbox_0.2.1-0.ubuntu-focal_amd64.deb'
sysbox is already the newest version (0.2.1-0.ubuntu-focal).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up sysbox (0.2.1-0.ubuntu-focal) ...
sysctl: cannot stat /proc/sys/kernel/unprivileged_userns_clone: No such file or directory
dpkg: error processing package sysbox (--configure):
 installed sysbox package post-installation script subprocess returned error exit status 255
Errors were encountered while processing:
 sysbox
E: Sub-process /usr/bin/dpkg returned an error code (1)

[ctalledo]

Thanks @felipecrs , good news that WSL2 is not on Linux 5.4. We will take a look to see if we can support WSL2 before the upcoming release (mid-Feb).

sysctl: cannot stat /proc/sys/kernel/unprivileged_userns_clone: No such file or directory

Looks like the installer is failing because it assumes it's on a Ubuntu machine, so it's looking for the /proc/sys/kernel/unprivileged_userns_clone sysctl. But that sysctl is not present in WSL2 apparently, so it fails.

[rodnymolina]

Hey @felipecrs, thanks for looking into this one.

You just uncovered an interesting scenario. You are attempting to install Sysbox over Ubuntu, but the kernel being utilized underneath is not an Ubuntu kernel, but a MSFT's one. That breaks a few assumptions we made in our installer. For example, the installer expects Ubuntu distros to have /proc/sys/kernel/unprivileged_userns_clone file present to prove that user-namespace's kernel feature is available, which is a Sysbox requirement.

I just verified that MSFT's kernel do enable unprivileged user-namespaces feature. However, they make use of a different approach followed by non-Ubuntu/Debian distros, which relies on the use of this file instead: /proc/sys/user/max_user_namespaces.

The fix for this one will be to add some logic to the Sysbox installer to identify the WSL2 setup and act accordingly to prevent the error you described above.

[felipecrs]

@rodnymolina sounds very promising! Hopefully, that's the only needed change.

[felipecrs]

Ok, more inputs to this topic:

• WSL2 Kernel got recently updated: 5.10.43.3
• Docker Desktop runs the containers in a sidecar distribution called docker-desktop, which is based on Alpine. So I believe this is where Sysbox would need to be installed to work, which sadly Sysbox is not yet compatible.

➜ wsl --list --verbose
  NAME                   STATE           VERSION
* Ubuntu                 Running         2
  docker-desktop-data    Running         2
  docker-desktop         Running         2

[rodnymolina]

@felipecrs, thanks for pointing that out, we will take it into account.

Now, please help me understand the use-case that you have in mind for Sysbox within WSL2 so that we prioritize this accordingly ...

Are you interested in the 'security' angle or you care more about the possibility of running 'docker-in-docker' setups? If it's the former, why is 'security' a concern within a personal development/testing environment? If it's the latter, can't docker-desktop + wsl2 run DIND with the typical 'privileged' container?

Thanks!

[felipecrs]

Oh, it's not for security reasons... I don't care about it in my personal environment.

My reasoning is: to have an environment in which I can test sysbox before deploying it in production. For example, I have an image which I use in a Jenkins to spawn disposable workers to run my builds (and Jenkins spawns them within a Kubernetes cluster).

https://github.com/felipecrs/jenkins-agent-dind

So, let's say I want to now make Jenkins use sysbox to deploy my workers. The first step is to configure my image (perhaps adding systemd, whatever). In order to do so, I need a development environment to edit the Dockerfile and call docker build and then docker run --runc sysbox to test it, which then my WSL2 enters in.

It's my dev env, which I would like to use for testing sysbox before deploying into production (Kubernetes).

---

But of course, I could spawn a new Ubuntu virtual machine here and use it as dev env instead. That's just harder and less productive than using WSL2.

[rodnymolina]

Thanks for the explanation @felipecrs, got it.

[felipecrs]

I'm not using Docker Desktop anymore, as they changed their licensing model. Instead, I installed the Docker Daemon directly on my Ubuntu 20.04 distro in WSL. So nevermind my concerns about running under Docker Desktop (which creates sidecars distributions and so).

I believe the first step would be to make it work without Docker Desktop, and then later, if possible, additionally support Docker Desktop.

The installation of the latest version of sysbox fails with:

#  https://github.com/microsoft/WSL2-Linux-Kernel
❯ uname -r
5.10.60.1-microsoft-standard-WSL2

❯ sudo apt install ./sysbox-ce_0.4.1-0.ubuntu-focal_amd64.deb -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'sysbox-ce' instead of './sysbox-ce_0.4.1-0.ubuntu-focal_amd64.deb'
sysbox-ce is already the newest version (0.4.1-0.ubuntu-focal).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up sysbox-ce (0.4.1-0.ubuntu-focal) ...

Your OS does not include the shiftfs module. Make sure to configure the container manager (e.g., Docker, CRI-O, etc) to use the Linux user-namespace when creating containers with Sysbox. Refer to Sysbox installation documentation for details.

sysctl: cannot stat /proc/sys/kernel/unprivileged_userns_clone: No such file or directory
dpkg: error processing package sysbox-ce (--configure):
 installed sysbox-ce package post-installation script subprocess returned error exit status 255
Errors were encountered while processing:
 sysbox-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)

Which is exactly the same as before, and @rodnymolina already elaborated.

Perhaps the sysbox installer could be made less aware of the distro which it was made for, so that a single .deb could be shipped that would work for either Debian, Ubuntu, and even the handle the situation which @rodnymolina described:

I just verified that MSFT's kernel do enable unprivileged user-namespaces feature. However, they make use of a different approach followed by non-Ubuntu/Debian distros, which relies on the use of this file instead: /proc/sys/user/max_user_namespaces.

Regardless, in no way I think this is a high priority issue.

[adamgit]

Now, please help me understand the use-case that you have in mind for Sysbox within WSL2 so that we prioritize this accordingly ...

Are you interested in the 'security' angle or you care more about the possibility of running 'docker-in-docker' setups? If it's the former, why is 'security' a concern within a personal development/testing environment? If it's the latter, can't docker-desktop + wsl2 run DIND with the typical 'privileged' container?

privileged container = so many problems, bugs, nightmares (avoiding all this mess is why we use sysbox/nestybox!) docker-deskttop + wsl2 = proprietary, confusing, hard to maintain.

sysbox + WSL2 = cross-platform development seamlessly for windows + linux + OSX teams.

The case I often run into is: teams that are using Docker to manage development. They have a set of dockerfiles, setup, etc - and that all needs wrapping in a container. I need to dockerize their docker. Without that wrap ... every developer is hand-maintaining everything in the local dev environment ("do I have the same version of Docker installed as my colleague? No? Oh! Damn! Now he/she can't build any more because I made a change that only works with my local version of Docker!") etc.

sysbox has enabled me to pick those setups up (done it a few times now), wrap them inside a suitably configured sysbox container, republish to the team - and suddenly all 'works for me' errors disappear. Now I can guarantee we're all start/stopping everything with the same versions, same dependencies, and same OS. It's also reduced the number of "works in development, but not in production" bugs.

I'm still evaluating sysbox for full production usage - running it locally for as much of my personal dev as possible, seeing if it works (so far, bar a few teething problems, it's done great).

...but then today I had to work on Windows, and sysbox wouldn't work :(. So for today's development I've fallen back to hand-maintained configuration and scripts (like coding in the dark ages).

[ctalledo]

Hi @adamgit,

Thank you very much for the useful feedback, and apologies for Sysbox not working on WSL2 yet.

We definitely want to enable Sysbox on WSL2, but have been swamped with other work. Let me sync-up with @rodnymolina to see if we can get this going and hopefully can deliver something in January 2022.

Regarding:

I need to dockerize their docker

Yes, this is one of the reasons we created Sysbox. We asked ourselves: why is it that only micro-service apps are Dockerized? Why aren't entire dev or test environments Dockerized too? The latter is really useful as you've seen (when done easily & securely), and it's the reason why Sysbox exists. Glad you are finding it useful!

[adamgit]

To be clear: I have no complaints! It's disappointing that WSL2 isn't supported yet - and it undermines my core use-case - but it works fantastically well for teams where we can guarantee everyone's already using linux.

But I've been pushing for fully containerized dev/test since early 2000's, so I can wait another 6 months for us to finally get there ;)...

On Wed, Dec 29, 2021 at 9:45 PM Cesar Talledo @.***> wrote:

Hi @adamgit https://github.com/adamgit,

Thank you very much for the useful feedback, and apologies for Sysbox not working on WSL2 yet.

We definitely want to enable Sysbox on WSL2, but have been swamped with other work. Let me sync-up with @rodnymolina https://github.com/rodnymolina to see if we can get this going and hopefully can deliver something in January 2022.

Regarding:

I need to dockerize their docker

Yes, this is one of the reasons we created Sysbox. We asked ourselves: why is it that only micro-service apps are Dockerized? Why aren't entire dev or test environments Dockerized too? The latter is really useful as you've seen, as it the reason why Sysbox exists. Glad you are finding it useful!

— Reply to this email directly, view it on GitHub https://github.com/nestybox/sysbox/issues/32#issuecomment-1002787412, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACHL6P5XGKZ452TPH25WTTUTN6QDANCNFSM4P754O5A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[felipecrs]

Out of curiosity, the WSL_DISTRO_NAME environment variable can be reliably used to detect whether running under a WSL2 environment or not.

[rodnymolina]

Got it, will check it out to see if this simplifies the WSL2 support effort.

[rodnymolina]

I just looked into this to asses the level-of-effort required to support WSL2. I started by fixing the unprivileged_userns_clone error described above. All good till there.

However, then I discovered a few other issues that need resolution:

• Looks like WSL2 does not support systemd (this is new for me), so we would need to extend Sysbox installer to accommodate the more traditional 'init' process manager. This would be a WSL specific requirement as I believe systemd is fully supported in most distros these days.

• configfs kernel module is not available in WSL2's kernel. We would need to understand how to add this module or relax this sysbox-mgr requirement.

• shiftfs module isn't provided by the current WSL2's 5.10.x kernel, so the user would need to build it manually till WSL2 kernel reaches 5.12 release (in which idmapped-mounts feature is available).

[arukiidou]

@rodnymolina

these issues resoleved.

• Looks like WSL2 does not support systemd
  • You can use a combination of Windows 11 and ubuntu 22.04. You also need to set up wsl.conf.
    • https://github.com/microsoft/WSL/issues/5126#issuecomment-1111957720
    • https://docs.microsoft.com/en-us/windows/wsl/wsl-config#boot-settings
• shiftfs module isn't provided by the current WSL2's 5.10.x kernel
  • WSL2's 5.15.x kernel is available on now.
  • https://github.com/microsoft/WSL2-Linux-Kernel/releases/tag/linux-msft-wsl-5.15.57.1
• configfs kernel module is not available in WSL2's kernel
  • Build a custom module. Then load it. The wsl2 kernel has no modules. there are only built-ins by default.
    • It was the most difficult of the three.

How did do it?

Here is an example of a kernel module build. see also: https://github.com/cilium/cilium/issues/17745#issuecomment-1004299480

• Get wsl kernel source.
  https://github.com/microsoft/WSL2-Linux-Kernel#build-instructions
• add settings to Microsoft/config-wsl

CONFIG_CONFIGFS_FS=m

• Build a custom module

# variables:
KCONFIG_CONFIG=Microsoft/config-wsl
KERNELRELEASE=5.15.57.1-microsoft-standard-WSL2

apt-get update && apt-get install -y build-essential flex bison dwarves libssl-dev libelf-dev python-is-python3 bc
make -j 7
make modules_install -j 7
make install -j 7

• Load a custom module

depmod
lsmod
modprobe configfs

What is my condition now?

wsl.conf

root@myconputer~# cat  /etc/wsl.conf
#[boot]
#command="/usr/libexec/wsl-systemd"

my kernel module

root@myconputer:~# lsmod
Module                  Size  Used by
configfs               45056  1

root@myconputer:/lib/modules# tree
.
└── 5.15.57.1-microsoft-standard-WSL2
    ├── build -> /builds/publics/WSL2-Linux-Kernel
    ├── kernel
    │   └── fs
    │       └── configfs
    │           └── configfs.ko
    ├── modules.alias
    ├── modules.alias.bin
    ├── modules.builtin
    ├── modules.builtin.alias.bin
    ├── modules.builtin.bin
    ├── modules.builtin.modinfo
    ├── modules.dep
    ├── modules.dep.bin
    ├── modules.devname
    ├── modules.order
    ├── modules.softdep
    ├── modules.symbols
    ├── modules.symbols.bin
    └── source -> /builds/publics/WSL2-Linux-Kernel

4 directories, 16 files

sysbox-mgr is enabled

root@myconputer~# systemctl list-units -t service --all | grep sysbox
#  sysbox-fs.service                      loaded    active   running sysbox-fs (part of the Sysbox container runtime)
#  sysbox-mgr.service                     loaded    active   running sysbox-mgr (part of the Sysbox container runtime)
#  sysbox.service                         loaded    active   running Sysbox container runtime

[arukiidou]

• sysbox is already available?
  • No, there are still a few more problems.

see: https://github.com/nestybox/sysbox/issues/439#issuecomment-1233313753

[rodnymolina]

@arukiidou, thanks for taking the time to make Sysbox work within WSL2 and for sharing the outcome of your effort.

The good news is that most of these issues are fixed now, so they will be all part of our upcoming release. In the meantime, you can build Sysbox from sources and try WSL2 once again.

These are the WSL2 limitations that I previously alluded to and the actions that we have carried out to mitigate them:

• Lack of systemd support in WSL2: We have recently enhanced the life-cycle management logic of Sysbox daemons to make systemd presence optional.

• configfs kernel module requirement: This is not a must-have requirement anymore (please refer to sysbox-fs' latest changes).

• shiftfs kernel module requirement: While it's true that this is becoming less of an issue now that idmapped-mounts feature is in the picture (i.e., kernel 5.15), we still encourage users to install shiftfs given that idmapped-mounts feature is still missing important capabilities (e.g., overlayfs mounts can't be id-mapped). However, even without shiftfs, and with some minor limitations, you should be able to make Sysbox work in your WSL2 setup.

• Finally, please be aware that we have recently fixed issue 439 too, so that shouldn't be a problem anymore.

Please let us know if you run into any other issue.

Thanks!

[felipecrs]

BTW Systemd is now available for WSL2:

https://devblogs.microsoft.com/commandline/systemd-support-is-now-available-in-wsl/

[audrenbdb]

configfs kernel module requirement: This is not a must-have requirement anymore

Running sudo ./scr/sysbox (from source installation instruction):

Could not load configfs kernel module. Exiting ...

And I can't proceed installation.

[rodnymolina]

@audrenbdb, I forgot to remove the configfs requirement from the scr/sysbox script -- typically we only use this one for testing, but it comes in handy in scenarios like yours where you're building from source, so you're right, this has to be fixed.

I'll submit a fix for this one tomorrow. In the meantime, just remove the configfs line from this file, and try again.

[arukiidou]

What about iptables-module? WSl2 has any modules, all built-in.

[ctalledo]

FYI, we should have this soon (before end-of-year), as we are currently working on adding support for Sysbox in WSL2 for Docker Desktop.