Using WSL, `build_in_docker` with pip requirements.txt fails because docker cannot mount /tmp

[lorengordon]

Description

Using WSL, build_in_docker with pip requirements.txt fails because docker cannot mount /tmp.

In package.py, the tempdir() function is using tempfile.mkdtemp(prefix=prefix) without specifying the dir argument. This means the tempdir will default to /tmp.

In WSL, docker currently cannot mount paths outside what is made available via the Docker Desktop file sharing settings. Typically, this means only the /c/ or /mnt/c/ paths are available. And only paths accessible from the Windows filesystem can be mounted.

This setup causes errors when package.py attempts to install the pip requirements.txt packages, because docker cannot find the requirements.txt file when mounting the tempdir at /tmp/....

> docker run --rm -v /tmp/terraform-aws-lambda-5ald8mpq:/var/task:z -v /home/loren/.ssh/known_hosts:/root/.ssh/known_hosts:z fe5786a6e5db /bin/sh -c 'python3 -m pip install --no-compile --prefix= --target=. --requirement=requirements.txt && chown -R 1000:1000 .'
WARNING: Running pip install with root privileges is generally not a good idea. Try `__main__.py install --user` instead.
Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
zip: Error during zip archive creation
Traceback (most recent call last):
  File ".terraform/modules/xxxxxx.lambda_layer/package.py", line 1138, in build_command
    bpm.execute(build_plan, zs, query)
  File ".terraform/modules/xxxxxx.lambda_layer/package.py", line 768, in execute
    with install_pip_requirements(query, pip_requirements) as rd:
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File ".terraform/modules/xxxxxx.lambda_layer/package.py", line 898, in install_pip_requirements
    pip_cache_dir=pip_cache_dir,
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'run', '--rm', '-v', '/tmp/terraform-aws-lambda-5ald8mpq:/var/task:z', '-v', '/home/loren/.ssh/known_hosts:/root/.ssh/known_hosts:z', 'fe5786a6e5db', '/bin/sh', '-c', 'python3 -m pip install --no-compile --prefix= --target=. --requirement=requirements.txt && chown -R 1000:1000 .']' returned non-zero exit status 1.

Versions

• Terraform: 0.14.7

Reproduction

Steps to reproduce the behavior:

module "lambda_layer" {
  source = "git::https://github.com/terraform-aws-modules/terraform-aws-lambda.git?ref=v1.44.0"

  create_layer = true

  layer_name  = "python-${random_string.this.result}"

  build_in_docker = true
  docker_file     = "${path.module}/layer/Dockerfile"
  docker_image    = "python-${random_string.this.result}"
  runtime         = "python3"
  source_path     = "${path.module}/layer"

  compatible_runtimes = [
    "python3.7",
    "python3.8"
  ]
}

Workaround

Thankfully, the default for the dir argument of tempfile.mkdtemp can be influenced by environment variables:

If dir is not None, the file will be created in that directory; otherwise, a default directory is used. The default directory is chosen from a platform-dependent list, but the user of the application can control the directory location by setting the TMPDIR, TEMP or TMP environment variables.

Therefore, this works as a workaround, presuming your PWD is available to docker as a shared path from the Windows filesystem: TMP=$(pwd) terraform apply

[jarinom]

I had the same issue with GitLab CI and setting the TMP environment variable to something else did the trick. For example defining variable TMP: "${CI_BUILDS_DIR}" for the job that runs terraform apply.

[mwiede]

thanks. building the lambda layer in docker also helped me to fix the GLIBC_2.18 not found issue similar to https://github.com/pyca/cryptography/issues/6390 I think the relevant piece here is to set the platform right. That was discussed in #346