Error running make compile-requirements

[alexgibson]

Description

I'm seeing the following error when running make compile-requirements

error: Broken virtualenv `/app/.venv`: `pyvenv.cfg` is missing

I have tried:

• Updating Docker desktop
• Running make pull
• Running make clean build

Neither seem to have made a difference

Environment

macOS 14.2 (23C64)

[alexgibson]

Full error:

(bedrock) alexgibson@agibson bedrock % make compile-requirements
/usr/local/bin/docker compose run --rm compile-requirements
+ export 'UV_CUSTOM_COMPILE_COMMAND=$ make compile-requirements'
+ UV_CUSTOM_COMPILE_COMMAND='$ make compile-requirements'
+ pip install -U uv
Collecting uv
  Downloading uv-0.2.37-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (37 kB)
Downloading uv-0.2.37-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.3/11.3 MB 3.2 MB/s eta 0:00:00
Installing collected packages: uv
Successfully installed uv-0.2.37

[notice] A new release of pip is available: 24.0 -> 24.2
[notice] To update, run: pip install --upgrade pip
+ rm -f 'requirements/*.txt'
+ uv pip compile --generate-hashes --no-strip-extras requirements/prod.in -o requirements/prod.txt
error: Broken virtualenv `/app/.venv`: `pyvenv.cfg` is missing
make: *** [compile-requirements] Error 2

[stevejalim]

Note: having DMed with Alex, I can't repro using Docker version 26.1.1, build 4cf5afa and Docker Compose version v2.27.0-desktop.2

[alexgibson]

(bedrock) alexgibson@agibson bedrock % docker --version         
Docker version 27.1.1, build 6312585
(bedrock) alexgibson@agibson bedrock % docker compose version   
Docker Compose version v2.29.1-desktop.1
(bedrock) alexgibson@agibson bedrock % 

[stevejalim]

For comparison, my output:

(bedrock) [steve] ~/Code/bedrock $ make compile-requirements                                                                                                                                                
/usr/local/bin/docker compose run --rm compile-requirements
+ export 'UV_CUSTOM_COMPILE_COMMAND=$ make compile-requirements'
+ UV_CUSTOM_COMPILE_COMMAND='$ make compile-requirements'
+ pip install -U uv
Collecting uv
  Downloading uv-0.2.37-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (37 kB)
Downloading uv-0.2.37-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.3/11.3 MB 13.8 MB/s eta 0:00:00
Installing collected packages: uv
Successfully installed uv-0.2.37

[notice] A new release of pip is available: 24.0 -> 24.2
[notice] To update, run: pip install --upgrade pip
+ rm -f requirements/dev.txt requirements/docs.txt requirements/prod.txt
+ uv pip compile --generate-hashes --no-strip-extras requirements/prod.in -o requirements/prod.txt
Resolved 134 packages in 19.87s
... [ continues without errors ] ...

[stevejalim]

The only difference I can see there is how my system unpacked rm -rf requirements/*.txt to rm -f requirements/dev.txt requirements/docs.txt requirements/prod.txt, but I don't see that'd be a factor in the virtualenv being broken.

@alexgibson It might be that there's a dodgy Docker image layer being cached somewhere. Could you remove your bedrock Docker images and make clean pull build test compile-requirements? That should clear the pipes

[alexgibson]

Could you remove your bedrock Docker images and make clean pull build test compile-requirements? That should clear the pipes

Everything builds ok, and tests pass, but still the same error :/

[stevejalim]

It's v strange how I can't repro this. Maybe the bedrock_build image you're using is unhappy, given that it's complaining that there are missing files

Stopping any running docker containers and then docker system prune --all --force --volumes should tear everything down and then make build will stand up all new things. Worth a shot!

[stevejalim]

@pmac Can you reproduce this on your local system?

[alexgibson]

Stopping any running docker containers and then docker system prune --all --force --volumes should tear everything down and then make build will stand up all new things. Worth a shot!

This still doesn't fix it for me unfortunately.

Is there a way for me to add/update Python dependencies that does not also require Docker?

[stevejalim]

Is there a way for me to add/update Python dependencies that does not also require Docker?

The preferred way is via Docker so we 100% get deps that will build in our production containers, but you can also just run the script on your local machine and the edge case that things are incompatible will get caught in CI anyway

You can run ./bin/compile-requirements.sh locally

[stevejalim]

@janbrasna As someone who is a Docker-based contributor, I'm wondering if you've ever hit this issue?

[janbrasna]

@stevejalim Never experienced. I've noticed this issue and would have chimed in if that sounded familiar — however I only consume the published images, not messing with deps locally unless trying out something stup*d (and that usually outside of containers as it's somewhat hard to change things in venvs there — I haven't studied too much how it may work without constant rebuilding). Also worth noting I'm running archaic Docker Desktop 4.15.0 (Engine 20.10.21, Compose v2.13.0) on x64 i.e. no compat mode, and definitely have several weeks old bedrock_build@b2a60a1c93d9 image, but already updated to use uv tho 🤷

(I'll try to run it for another data point the next time I heat up the engine… I spin up a separate unprivileged account just for that.)

EDIT: Nope I'm good. It's running from userspace though, and no venv is being used in the workdir itself so that's probably different from your case.