bug: Python is being upgraded to 3.12 which is blocking the install process

[oc-liz-conlan]

We're seeing this (started around 3 days ago) trying to run [email protected] in our GitHub Actions flow:

Prepare all required actions
Run ./../../_actions/LocalStack/setup-localstack/v0.2.3/tools
Run pip install --upgrade pip
  pip install --upgrade pip
  which localstack > /dev/null || pip install localstack
  if [ "$INSTALL_AWSLOCAL" = true ]; then
    which awslocal > /dev/null || pip install awscli-local[ver1]
  fi
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    GH_ACTION_ROOT: ./../../_actions/LocalStack/setup-localstack/v0.2.3
    SERVICES: s3
    INSTALL_AWSLOCAL: true
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
Error: Process completed with exit code 1.

It was previously packaging python3.10 and working fine but the changes to pip now that it's packaging 3.12 are preventing the script from completing 😬

[alexrashed]

Thanks, @oc-liz-conlan for raising the issue. I'm having a bit of a hard time understanding exactly what it is that you changed. It sounds like 3 days ago you changed your runner configuration such that it now does not work anymore? If this is the case, what are the exact changes you did? What changed 3 days ago? The action itself didn't see any changes lately.

[oc-liz-conlan]

As far as I'm aware we haven't changed our runner at all 🤔

Maybe there's something odd happening upstream on our side that's unexpectedly upgraded python and introduced a breaking change for pip install. I'll dig a bit more and hopefully close the ticket tomorrow

[oc-liz-conlan]

Figured it out. I was half right but hadn't realised that there had been a change in a different part of our system that was unconditionally upgrading python to latest 🤦‍♀️ (I'll find a workaround for that)

However… Python 3.12 has introduced a breaking change to the way packages are installed into system. It now needs to be passed --break-system-packages when a system-wide package is being added intentionally otherwise it will stop with a warning (which breaks the Setup/LocalStack install :sob:) I'll defer to Dev.to for the full explanation

So there is an issue to consider but not quite the one I thought it was

Happy for the ticket to be closed as I've misdescribed the problem (sorry!)

[indeedhat]

Just to add a little more info into this i am getting the same error on a pipeline that hasn't been updated in 6 months.

I did a test where i created a bare minimum pipeline https://github.com/indeedhat/localstack-ci-test and that worked just fine.

Sorry i can't share the pipeline its failing on, its on a private work repo but both are using ubuntu-latest and the one that is failing doesn't explicitly updated python, in fact checking the log output the only occurrences of 'python' is in the error message itself.

[alexrashed]

Unfortunately, this still means that we cannot fully pin down the issue, right?

@indeedhat you are saying that you are using ubuntu-latest where you see the issue in your private repo, but can't reproduce the issue in your public repo? Are there any other specifics of your runner?

@oc-liz-conlan I'm happy to continue tracking down the issue here in this ticket if there's something we should do here in this repo, but I'm having a hard time actually distilling a takeaway from what we know right now. Are you sure you are using a default GitHub runner? In this case I would be surprised not to find any other references on the interwebs of GitHub actions failing due to this change (didn't search for too long yet though)...

[indeedhat]

the actions we are using:

• actions/checkout@v3
• shivammathur/setup-php@v2
• docker/setup-buildx-action@v2
• docker/login-action@v2
• docker/build-push-action@v5
• codecov/codecov-action@v3

None of the on the face of it seem to update python

we also have a postgres service using postgres:16-alpine

All told its a pretty basic action. Im kind of wondering if github are rolling out updates to ubuntu-latest and not all repo's get the new version running python 3.12

UPDATE: seems like its a rollout thing... i copied over by basic test pipeline to the same private repo that was failing and it also fails.

For now im side stepping the issue by running this before the setup localstack action:

- name: Revert to python 3.10
  uses: actions/setup-python@v5
  with:
    python-version: '3.10'

[alexrashed]

It seems like GitHub is currently preparing an upgrade of the ubuntu-latest image and there might be some A-B testing involved? https://github.com/actions/runner-images/issues/10636

@indeedhat Maybe you could try to explicitly set runs-on: ubuntu-22.04? This could maybe help to build a reliable reproducer for the case?

[indeedhat]

good call, ubuntu-22.04 runs just fine, ubuntu-24.04 gives us the error: https://github.com/indeedhat/localstack-ci-test/actions/runs/11126549135/job/30916723607

[alexrashed]

In my opinion, it's a best practice in GitHub workflows to use actions which install system-level packages, so in my opinion this might just be an issue with the upcoming runnner configuration? The Python installation should just not be marked as externally managed? Would you mind bringing the issue up in https://github.com/actions/runner-images/issues/10636?

[oc-liz-conlan]

@indeedhat aha! my pipeline is setting runs-on: ubuntu-latest so it looks like I can set it to ubuntu-22.04 (as that packages 3.10) instead for now

In case it helps anyone, the relevant bit of my (lightly redacted) GitHub workflow is:

jobs:
  rspec:
    runs-on: ubuntu-latest

    steps:
      - name: Git checkout
        uses: actions/checkout@v3

      - name: Set up Ruby
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: '2.7.8'

      - name: Start LocalStack
        uses: LocalStack/[email protected]
        with:
          image-tag: 'latest'
          install-awslocal: 'true'
        env:
          SERVICES: 's3'

(It's doing nothing fancy at all, no custom runners. I've spared you my generic bundle install / rspec setup code which comes after so it's not being reached at the moment)

[oc-liz-conlan]

@alexrashed comment added! https://github.com/actions/runner-images/issues/10636#issuecomment-2386131516

[alexrashed]

@oc-liz-conlan @indeedhat The migration to the new runners should be complete now (according to their announcement blog post). Could you verify if you still see this issue? As far as I can see, we haven't been facing this issue on our end, and I would hope that GitHub fixed it before using the new runner globally...

[indeedhat]

ive tested it on both ubuntu-24.04 and ubuntu-latest and in both cases it works just fine. Thanks.

[alexrashed]

Thanks for reporting back! This means it was an issue on the runner config. I'll closing this one for now. Thanks for the help to everyone involved! :)