Cannot run `cross` in Docker in Docker in WSL

[bombsimon]

Continuation of https://github.com/cross-rs/cross/issues/260#issuecomment-1135568872 and https://github.com/cross-rs/cross/issues/260#issuecomment-1141224292.

Description of the problem

Running cross from a Docker container (e.g. Docker in Docker) that runs in WSL does not work, ~~it seems like the root cause is how Docker and volumes are created and mounted in WSL but this is so far just a guess~~.

Running with CROSS_DOCKER_IN_DOCKER=true (which I should have to) will discover a volume path that is not mounted in the current container.

Runing with CROSS_DOCKER_IN_DOCKER=false won't work because mounting a directory from a Docker container to another does not work afaik.

Setup

• Windows 11
• WSL2
• Ubuntu 20.04
• Rancher Desktop 1.3.0
• cross 0.2.1
• cargo 1.61.0

How to reproduce

Start build container that includes Docker

docker run --rm -it --entrypoint bash \
  -e CROSS_DOCKER_IN_DOCKER=true \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gcr.io/cloud-builders/docker

Install Rust and cross, create project

apt update && apt install build-essential -y
curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain stable
source $HOME/.cargo/env
cargo install cross
cargo init hello-world && cd hello-world

Try to build with cross

cross build
info: downloading component 'rust-src'
info: installing component 'rust-src'
sh: 1: cargo: not found

Quicker repro without cross

Start DinD container and create file

docker run --rm -it --entrypoint /bin/sh -v /var/run/docker.sock:/var/run/docker.sock docker:latest
apk add --update jq
mkdir test
echo "Hello, World" > test/file.md

Mount the MergedDir directory

This will work on macOS and Linux but not WSL.

docker run --rm -it \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v $(docker inspect $HOSTNAME | jq -r .[0].GraphDriver.Data.MergedDir)/test:/workdir \
  docker:latest cat /workdir/file.md

Troubleshooting root cause

Show more

UPDATE: This seems to be incorrect and irrelevant

I'm not very familiar with WSL and Docker but I noticed that if I start the container like in the first step from WSL, install jq and look at the same path as what the code found here does, I get:

docker inspect $HOSTNAME | jq -r .[0].GraphDriver.Data.MergedDir
/var/lib/docker/overlay2/7c8b6ec02033086d5f6de2b10b984991d92bddd2c58e893216dfef6a67a0ce3a/merged

stat /var/lib/docker
stat: cannot stat '/var/lib/docker': No such file or directory

I guess this is not expected since the output from cross build -v refers to this directory for the whole toolchain:

CROSS_DOCKER_IN_DOCKER=true cross build -v
+ "rustc" "--print" "sysroot"
+ "rustup" "toolchain" "list"
+ "rustup" "target" "list" "--toolchain" "stable-x86_64-unknown-linux-gnu"
+ "rustup" "component" "list" "--toolchain" "stable-x86_64-unknown-linux-gnu"
+ "/usr/bin/docker" "run" "--userns" "host" "-e" "PKG_CONFIG_ALLOW_CROSS=1" "--rm" "--user" "0:0" "-e" "XARGO_HOME=/xargo" "-e" "CARGO_HOME=/cargo" "-e" "CARGO_TARGET_DIR=/target" "-e" "USER=root" "-e" "CROSS_RUNNER=" "-v" "/var/lib/docker/overlay2/7c8b6ec02033086d5f6de2b10b984991d92bddd2c58e893216dfef6a67a0ce3a/merged/root/.xargo:/xargo:Z" "-v" "/var/lib/docker/overlay2/7c8b6ec02033086d5f6de2b10b984991d92bddd2c58e893216dfef6a67a0ce3a/merged/root/.cargo:/cargo:Z" "-v" "/cargo/bin" "-v" "/var/lib/docker/overlay2/7c8b6ec02033086d5f6de2b10b984991d92bddd2c58e893216dfef6a67a0ce3a/merged/hello-world:/project:Z" "-v" "/var/lib/docker/overlay2/7c8b6ec02033086d5f6de2b10b984991d92bddd2c58e893216dfef6a67a0ce3a/merged/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:/rust:Z,ro" "-v" "/var/lib/docker/overlay2/7c8b6ec02033086d5f6de2b10b984991d92bddd2c58e893216dfef6a67a0ce3a/merged/hello-world/target:/target:Z" "-w" "/project" "-i" "-t" "rustembedded/cross:x86_64-unknown-linux-gnu-0.2.1" "sh" "-c" "PATH=$PATH:/rust/bin cargo build -v"
sh: 1: cargo: not found

I would guess this has to do with how Docker Desktop and Rancher Desktop sets up data volumes separate from each other and separate from WSL:

wsl --list -v
  NAME                    STATE           VERSION
* Ubuntu-20.04            Running         2
  rancher-desktop         Running         2
  rancher-desktop-data    Running         2

[bombsimon]

Ok I was probably wrong regarding expectations of finding /var/lib/docker even if it's mentioned in the GraphDriver output. Since this is what cross mounts and it works on other OSes, I thought it was the root cause. However, I booted up a Linux VM to try to reproduce this and there too I cannot find /var/lib/docker in the Docker container but it still works if I follow the steps above to reproduce... 🤔

root@c80df52385bc:/hello-world# cross build -v
+ "rustc" "--print" "sysroot"
+ "rustup" "toolchain" "list"
+ "rustup" "target" "list" "--toolchain" "stable-x86_64-unknown-linux-gnu"
+ "rustup" "component" "list" "--toolchain" "stable-x86_64-unknown-linux-gnu"
+ "rustup" "component" "add" "rust-src" "--toolchain" "stable-x86_64-unknown-linux-gnu"
info: downloading component 'rust-src'
info: installing component 'rust-src'
+ "/usr/bin/docker" "run" "--userns" "host" "-e" "PKG_CONFIG_ALLOW_CROSS=1" "--rm" "--user" "0:0" "-e" "XARGO_HOME=/xargo" "-e" "CARGO_HOME=/cargo" "-e" "CARGO_TARGET_DIR=/target" "-e" "USER=root" "-e" "CROSS_RUNNER=" "-v" "/var/lib/docker/overlay2/b738b0420860126166e5b690beed1707a5c188debf66652ae271860785047119/merged/root/.xargo:/xargo:Z" "-v" "/var/lib/docker/overlay2/b738b0420860126166e5b690beed1707a5c188debf66652ae271860785047119/merged/root/.cargo:/cargo:Z" "-v" "/cargo/bin" "-v" "/var/lib/docker/overlay2/b738b0420860126166e5b690beed1707a5c188debf66652ae271860785047119/merged/hello-world:/project:Z" "-v" "/var/lib/docker/overlay2/b738b0420860126166e5b690beed1707a5c188debf66652ae271860785047119/merged/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:/rust:Z,ro" "-v" "/var/lib/docker/overlay2/b738b0420860126166e5b690beed1707a5c188debf66652ae271860785047119/merged/hello-world/target:/target:Z" "-w" "/project" "-i" "-t" "rustembedded/cross:x86_64-unknown-linux-gnu-0.2.1" "sh" "-c" "PATH=$PATH:/rust/bin cargo build -v"
   Compiling hello-world v0.1.0 (/project)
     Running `rustc --crate-name hello_world --edition=2021 src/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 -C metadata=24ded0f759cf8b5b -C extra-filename=-24ded0f759cf8b5b --out-dir /target/debug/deps -C incremental=/target/debug/incremental -L dependency=/target/debug/deps`
    Finished dev [unoptimized + debuginfo] target(s) in 0.41s

stat /var/lib/docker/
stat: cannot stat '/var/lib/docker/': No such file or directory

And also confirming that just running the following will work:

docker run --rm -it \
  -v "/var/lib/docker/overlay2/b738b0420860126166e5b690beed1707a5c188debf66652ae271860785047119/merged/root/.cargo:/cargo:Z" \
  rustembedded/cross:x86_64-unknown-linux-gnu-0.2.1 stat /cargo/bin/cargo
  File: '/cargo/bin/cargo'
  Size: 15690192        Blocks: 30648      IO Block: 4096   regular file
Device: 2ch/44d Inode: 792675      Links: 12
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-05-30 15:09:20.657129255 +0000
Modify: 2022-05-30 15:09:01.355426167 +0000
Change: 2022-05-30 15:09:01.355426167 +0000
 Birth: -

So I guess I'll have to read up on the whole /var/lib/docker/overlay2 thing, also haven't used DinD too much so this is new to me.

[bombsimon]

Ok so I read up a bit and the whole /var/lib/docker thing is obviously in the VM that runs Docker so that would be the rancher-desktop-data distro for Windows and I think Lima for macOS.

The -v option is just a flag back to the Docker daemon host which communicates with the VM where these files would be found.

The reason why this works on macOS, Linux and Windows but not WSL is because the docker.sock is not mounted the same way. So if I check where the docker.sock actually exists in Windows from git bash I get:

$ cygpath.exe -w /var/run/docker.sock\;C/
C:\Program Files\git\var\run\docker.sock;C\

This is what I use when I from Windows run:

-v //var/run/docker.sock:/var/run/docker.sock

That is not the same as /var/run/docker.sock in WSL. I guess I would have to mount something like this for it to work in WSL:

-v "/mnt/c/Program Files/git/var/run/docker.sock\;C/:/var/run/docker.sock"

However I don't really know how to properly mount this socket properly given the magic going on with the appended ;C etc.

Either way I don't think this is an issue with cross. Once I figure out how to mount the same socket as used by the host OS this should start to work automatically. I'll close this for now and if/when I figure this out I'll post an update.

[Alexhuszagh]

This is reproducing, but has some interesting quirks. I'll be commenting here and trying to debug. Current information:

$ docker run --rm -it --entrypoint bash \
  -e CROSS_DOCKER_IN_DOCKER=true \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gcr.io/cloud-builders/docker
$ apt update && apt install build-essential -y
curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain stable
$ source $HOME/.cargo/env
$ cargo install cross --debug
$ cargo init hello-world && cd hello-world
$ ls -la
total 24
drwxr-xr-x 4 root root 4096 Jul 18 16:14 .
drwxr-xr-x 1 root root 4096 Jul 18 16:14 ..
drwxr-xr-x 6 root root 4096 Jul 18 16:14 .git
-rw-r--r-- 1 root root    8 Jul 18 16:14 .gitignore
-rw-r--r-- 1 root root  180 Jul 18 16:14 Cargo.toml
drwxr-xr-x 2 root root 4096 Jul 18 16:14 src
$ docker run -it --rm -v /var/lib/docker/overlay2/$ID/"$PWD":/project -w /project ubuntu:20.04 bash -c "ls -la"
total 12
drwxr-xr-x 3 root root 4096 Jul 18 15:46 .
drwxr-xr-x 1 root root 4096 Jul 18 16:14 ..
drwxr-xr-x 2 root root 4096 Jul 18 15:46 target
$ mkdir testing
$ docker run -it --rm -v /var/lib/docker/overlay2/$ID/"$PWD":/project -w /project ubuntu:20.04 bash -c "ls -la"
total 12
drwxr-xr-x 3 root root 4096 Jul 18 15:46 .
drwxr-xr-x 1 root root 4096 Jul 18 16:14 ..
drwxr-xr-x 2 root root 4096 Jul 18 15:46 target

Note how the target directory is mounted even though it doesn't exist, and the other directories are not. Using any other value in docker inspect $HOSTNAME doesn't seem to work, including the hostname itself.