Document how to enable docker environment for docker-in-docker setup

[chris-stetter]

Describe the bug I am trying to use the new environments feature to run a test in a docker container. I am running Pants itself in a docker container (VSCode devcontainer). Running the test fails:

vscode ➜ /workspaces/pants-2.15 (master ✗) $ ./pants test tests/example_test.py 
08:55:13.90 [ERROR] 1 Exception encountered:

Engine traceback:
  in `test` goal
  in Run Pytest - (environment:linux_docker, tests/example_test.py)

ProcessExecutionFailure: Process 'Extract environment variables from the Docker image python:3.8-bullseye' failed with exit code 126.
stdout:
OCI runtime exec failed: exec failed: unable to start container process: chdir to cwd ("/pants-sandbox/pants-sandbox-QvFrxo") set in config.json failed: no such file or directory: unknown

stderr:



Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.

Check out the code here https://github.com/chris-stetter/pants-2.15. I think the reason is simply that the files that are being mounted to the docker test container are not present on the host, but only in my container where I am running Pants. I can fix it the following way:

vscode ➜ /workspaces/pants-2.15 (master ✗) $ git diff
diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json
index df25fb1..c2b48aa 100644
--- a/.devcontainer/devcontainer.json
+++ b/.devcontainer/devcontainer.json
@@ -1,7 +1,8 @@
 {
        "name": "Python 3",
        "image": "mcr.microsoft.com/vscode/devcontainers/python:3.8-bullseye",
-       "mounts": ["source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind,consistency=cached"],
+       "mounts": ["source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind,consistency=cached",
+                          "source=/tmp,target=/tmp,type=bind,consistency=cached"],
 
        // Configure tool-specific properties.
        "customizations": {
diff --git a/pants.toml b/pants.toml
index 326451d..b801c98 100644
--- a/pants.toml
+++ b/pants.toml
@@ -1,5 +1,6 @@
 [GLOBAL]
 pants_version = "2.15.0rc0"
+named_caches_dir = "/tmp/named_caches"
 
 backend_packages = [
   'pants.backend.python',

That way, all files are also present on the host file system.

Pants version 2.15.0rc0

OS Linux

Additional info Should something like this be added to the docs?

[Eric-Arellano]

cc @benjyw, you're currently debugging this exact same issue

[benjyw]

Interesting. I think my error was different - there was no docker-in-docker, but an issue with running on Linux ARM64 that manifested the same error message.

[stuhood]

OS Linux

@chris-stetter : Can you confirm that your host (outermost) is Linux as well?

[stuhood]

AFAICT, this is tangled up in the question of Docker-in-Docker vs Docker-from-Docker, as differentiated on https://code.visualstudio.com/remote/advancedcontainers/use-docker-kubernetes.

Your example repository uses Docker-from-Docker, meaning that it connects to a docker instance running on your host. Thus, unless /tmp is accessible from both the host and the dev container, that docker instance running on the host will not have access to the sandboxes which have been created inside of the dev container, which are then supposed to be mounted into the innermost container. This problem is mentioned on the linked page.

I think that the need to declare that extra bind mount is unavoidable with Docker-from-Docker, so I think that expanded documentation might be the answer. Is there any particular reason that you chose Docker-from-Docker rather than Docker-in-Docker?

[stuhood]

I think that the need to declare that extra bind mount is unavoidable with Docker-from-Docker, so I think that expanded documentation might be the answer. Is there any particular reason that you chose Docker-from-Docker rather than Docker-in-Docker?

Except perhaps by using docker volumes, rather than by bind mounting temporary directories. But I expect that that would be slower.

EDIT: Nevermind. That wouldn't help, because the container that populated the volume would still need access to the inputs.

[chris-stetter]

Is there any particular reason that you chose Docker-from-Docker rather than Docker-in-Docker?

Nothing particular, but using a docker-compose file with other services. Not sure if that works with Docker-in-Docker.

[lecardozo]

I'm experiencing a similar issue (more on this Slack thread). However, my setup is a bit different as I run Pants (2.16) directly on the host (OSX, M1 arch) where docker is running. Any thoughts on how to debug that? Thanks!

[lecardozo]

Just getting back here after finally figuring out what the issue was!

I'm using colima as container runtime for the M1 and, by default, it only mounts the ~ and /tmp/colima inside the VM where the container runtime is executed. Since pants' temporary working directory that is shared with the container lives on /private/var/folders, I had to set that additional mount on colima's config explicitly. After that, things started working as expected. 🎉

[benjyw]

Thanks for the update!