Bug Report: cimg/python:x.y is unusable with venv caching

[christophehenry]

Describe the bug Caching venv si common practice as documented. However, on cimg/python, python -m venv venv produces a virtual environment with a symlink to a fixed x.y.z python version (/home/circleci/.pyenv/versions/3.11.10/bin/python for instance on cimg/python:3.11.

This causes . venv/activate to fail with Error: [Errno 2] No such file or directory: 'venv/bin/python' as soon as the image is updated with a new minor Python version.

To Reproduce Using the following config.yml:

jobs:
  build:
    docker:
      - image: cimg/python:3.11.4
    steps:
      - checkout
      - restore_cache:
          keys:
            - << pipeline.parameters.cache-version >>-{{ checksum "Pipfile.lock" }}
            - << pipeline.parameters.cache-version >>-
      - run:
          name: install dependencies
          command: |
            python -m venv /tmp/venv
            . /tmp/venv/bin/activate
            pip install pipenv
            pipenv install --dev
      - save_cache:
          paths:
            - /tmp/venv
          key: << pipeline.parameters.cache-version >>-{{ checksum "Pipfile.lock" }}

Change the used image from cimg/python:3.11.4 to cimg/python:3.11 and the CI should fail with the No such file or directory error.

Expected behavior Caching the venv should not make the CI to suddenly fail as soon as the Python minor changes when using cimg/python:x.y.

Workarounds Use cimg/python:x.y.z instead of cimg/python:x.y wich is inconveniant.

[christophehenry]

IDK how to solve that, though. I couldn't find a way to make python -m venv to use /home/circleci/.pyenv/shims/ rather than /home/circleci/.pyenv/versions/3.11.10/bin/.

[cclauss]

https://pipenv.pypa.io/en/latest/installation.html#installing-packages-for-your-project

pipenv install requests will automatically create a virtualenv (which is different that Python's builtin venv).

[christophehenry]

I'm sorry, how does that solve the problem?

[cclauss]

      - run:
          name: install dependencies
          command: |
-           python -m venv /tmp/venv
-           . /tmp/venv/bin/activate
-           pip install pipenv
+           pip install --user pipenv
            pipenv install --dev
+           # Look at the logs to see where the `virtualenv` is being created.
      - save_cache:
          paths:
-           - /tmp/venv
+           # We need something related to `virtualenv` here (NOT `venv`)
          key: << pipeline.parameters.cache-version >>-{{ checksum "Pipfile.lock" }}
+         # We need `pipenv run` to access the installed deps.
+         run: pipenv run ABC

[christophehenry]

That won't work either. pipenv also takes Pyenv's Python as the reference implementation:

$ pipenv install requests
Creating a virtualenv for this project...
Pipfile: /home/circleci/repo/Pipfile
Using /home/circleci/.pyenv/versions/3.11.10/bin/python (3.11.10) to create virtualenv...
  Creating virtual environment...created virtual environment CPython3.11.10.final.0-64 in 848ms
  creator CPython3Posix(dest=/home/circleci/.local/share/virtualenvs/repo-eQF46Ow3, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/circleci/.local/share/virtualenv)
    added seed packages: pip==24.2, setuptools==74.1.2, wheel==0.44.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

Successfully created virtual environment!
Virtualenv location: /home/circleci/.local/share/virtualenvs/repo-eQF46Ow3

$ ll ~/.local/share/virtualenvs/repo-eQF46Ow3/bin
python -> /home/circleci/.pyenv/versions/3.11.10/bin/python

And again: if I'm caching that virtualenv, it'll break as soon as the image changed the minor version.