Docker in Docker doesn't seem to find / forward the volume mounts

[nuket]

Checklist

• [X] I've looked through the issues and pull requests for similar reports

Describe your issue

Using a clean Ubuntu Server 22.04.2 LTS system, running cross inside of one Docker container gives me the same error Build fails with sh: 1: cargo: not found listed in https://github.com/cross-rs/cross/issues/260.

If /var/lib/docker is not bind mounted to the outer Docker instance, when cross goes to forward some of those mounts, there is nothing to forward to the inner Docker instance:

rustuser@rust-build:~/workspace$ cross build -vv --target aarch64-unknown-linux-gnu
+ cargo metadata --format-version 1 --filter-platform aarch64-unknown-linux-gnu
+ rustc --print sysroot
+ rustup toolchain list
+ rustup target list --toolchain 1.70.0-x86_64-unknown-linux-gnu
+ rustup component list --toolchain 1.70.0-x86_64-unknown-linux-gnu
+ /usr/bin/docker
+ /usr/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=rustuser' -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/builddir/library:/var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/builddir/library --rm --user 1000:1000 -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/.xargo:/xargo:z -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/opt/cargo:/cargo:z -v /cargo/bin -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace:/var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace:z -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu:/rust:z,ro -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/target:/target:z -w /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace -i -t ghcr.io/cross-rs/aarch64-unknown-linux-gnu:0.2.5 sh -c 'PATH=$PATH:/rust/bin cargo build -vv --target aarch64-unknown-linux-gnu'
sh: 1: cargo: not found
+ rustup component list --toolchain 1.70.0-x86_64-unknown-linux-gnu

If you run the container above and drop into a shell, you can examine the paths available in /var/lib and see what is missing.

rustuser@rust-build:~/workspace$  /usr/bin/docker run --userns host -e 'PKG_CONFIG_ALLOW_CROSS=1' -e 'XARGO_HOME=/xargo' -e 'CARGO_HOME=/cargo' -e 'CARGO_TARGET_DIR=/target' -e 'CROSS_RUNNER=' -e TERM -e 'USER=rustuser' -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/builddir/library:/var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/builddir/library --rm --user 1000:1000 -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/.xargo:/xargo:z -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/opt/cargo:/cargo:z -v /cargo/bin -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace:/var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace:z -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/usr/local/rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu:/rust:z,ro -v /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace/target:/target:z -w /var/lib/docker/overlay2/usdd8fbv8j2xwmrqyhw5v14b3/merged/home/rustuser/workspace -i -t ghcr.io/cross-rs/aarch64-unknown-linux-gnu:0.2.5 sh
$ ls
target
$ id
uid=1000 gid=1000 groups=1000
$ ls /
bin  boot  cargo  common.sh  dev  etc  home  lib  lib.sh  lib64  linux-image.sh  linux-runner  media  mnt  opt  proc  qemu  root  run  rust  sbin  srv  sys  target  tmp  usr  var  xargo
rustuser@rust-build:~/workspace$ ls -l /var/lib
total 48
drwxr-xr-x 1 root root 4096 Jun 28 14:25 apt
drwxr-xr-x 1 root root 4096 Jun 28 14:26 dpkg
drwxr-xr-x 2 root root 4096 Feb 22 10:51 git
drwxr-xr-x 2 root root 4096 Apr  2 13:55 misc
drwxr-xr-x 2 root root 4096 Jun 12 02:00 pam
drwxr-xr-x 2 root root 4096 Jun 13 05:29 python
drwxr-xr-x 3 root root 4096 Jun 28 14:25 sudo
drwxr-xr-x 1 root root 4096 Jun 13 05:29 systemd
drwxr-xr-x 3 root root 4096 Jun 13 05:29 ucf
rustuser@rust-build:~/workspace$ ls -l /var/lib/docker
ls: cannot access '/var/lib/docker': No such file or directory

I don't have a solution to this yet. It should be possible to bind mount /var/lib/docker, but the root:root permissions will be an issue.

System Info

Host system:

$ uname -a
Linux boombox 5.15.0-72-generic #79-Ubuntu SMP Wed Apr 19 08:22:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:        22.04
Codename:       jammy

$ docker --version
Docker version 24.0.2, build cb74dfc

Outer Docker is rust:latest (debian:11, 11.7, bullseye, bullseye-20230612):

$ docker --version
Docker version 20.10.5+dfsg1, build 55c4c88

$ cross --version
cross 0.2.5
[cross] note: Falling back to `cargo` on the host.
cargo 1.70.0 (ec8a8a0ca 2023-04-25)

Inner Docker is the aarch64-unknown-linux-gnu image.

What target(s) are you cross-compiling for?

aarch64-unknown-linux-gnu

Which operating system is the host (e.g computer cross is on) running?

• [ ] macOS
• [ ] Windows
• [X] Linux / BSD
• [ ] other OS (specify in description)

What architecture is the host?

• [X] x86_64 / AMD64
• [ ] arm32
• [ ] arm64 (including Mac M1)

What container engine is cross using?

• [X] docker
• [ ] podman
• [ ] other container engine (specify in description)

cross version

cross 0.2.5

Example

https://github.com/cross-rs/cross/issues/260#issuecomment-1611735744

Examining the Docker command line that is run by cross, it finds the /var/lib/docker mounts, I believe by querying the docker.sock, but because the outer Docker container does not mount /var/lib/docker, the inner Docker can't then access any of those mounts.

It would be an issue anyways as the outer Docker container marks all mounts root:root in /var/lib/docker.

Additional information / notes

I'm trying to think of workarounds.

One of which would be to run rustup in my non-admin account in the host system, and then run the aarch64-unknown-linux-gnu container.

Another workaround might be Cross.toml offering a way to specify what folders to mount for which tools.

Another workaround might be to use a Docker managed volume for the tools and cross.

Final option would be to reproduce the mounts in a docker-compose.yml and reuse the cross-rs containers from the host system.