`batou 2.5.0b3 has requirement setuptools>=66.1.0, but you have setuptools 65.5.0.`

[Ma27]

Given

• Python 3.11.9 installed from nixpkgs (with setuptools 69.5.1.post0)
• appenv at 2aee66bb3bd71eb2327afd654b7b09d862c39db3
• batou 2.5.0b3

I get the following error when running ./appenv python:

Existing envdir not consistent, deleting
Creating venv ...
Ensuring pip ...
Installing ...
['/Users/ma27/Projects/deployment-batou/./.appenv/781ad8b9/bin/python', '-m', 'pip', 'check'] returned with exit code 1
batou 2.5.0b3 has requirement setuptools>=66.1.0, but you have setuptools 65.5.0.

Traceback (most recent call last):
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 50, in cmd
    return subprocess.check_output(c, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/pha8krh62hzq1z1j1hhmhsxn0kw9v439-python3-3.11.9/lib/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/pha8krh62hzq1z1j1hhmhsxn0kw9v439-python3-3.11.9/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/ma27/Projects/deployment-batou/./.appenv/781ad8b9/bin/python', '-m', 'pip', 'check']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 609, in <module>
    main()
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 603, in main
    appenv.meta()
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 365, in meta
    args.func(args, remaining)
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 506, in python
    self.run('python', remaining)
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 368, in run
    self.prepare()
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 453, in prepare
    pip(env_dir, ["check"])
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 62, in pip
    return python(path, ["-m", "pip"] + c, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 58, in python
    return cmd([os.path.join(path, "bin/python")] + c, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ma27/Projects/deployment-batou/./appenv", line 54, in cmd
    raise ValueError(e.output.decode("utf-8", "replace"))
ValueError: batou 2.5.0b3 has requirement setuptools>=66.1.0, but you have setuptools 65.5.0.

Workaround for me was nix shell nixpkgs#python312.

---

I assume this is because appenv bootstraps itself from https://www.python.org/ftp/python/3.11.9/Python-3.11.9.tgz which has a setuptools 65.5.0 bundled.

If I understand the issue correctly, the bump of setuptools to 66.1 breaks most of the Python versions supported by batou because of the way appenv works.

cc @zagy @elikoga

[Ma27]

Hmm, with Python 3.12 and batou 2.5b3 I get

      Exception: No module named 'botocore.vendored.six.moves'
Traceback (simplified, most recent call last):
  File "/Users/ma27/Projects/deployment-batou/components/redacted/component.py", line 11, in <module>
    import batou_ext.s3
  File "/Users/ma27/Projects/deployment-batou/.appenv/8fa9d3b1/lib/python3.12/site-packages/batou_ext/s3.py", line 25, in <module>
    import boto3
  File "/Users/ma27/Projects/deployment-batou/.appenv/8fa9d3b1/lib/python3.12/site-packages/boto3/__init__.py", line 16, in <module>
    from boto3.session import Session
  File "/Users/ma27/Projects/deployment-batou/.appenv/8fa9d3b1/lib/python3.12/site-packages/boto3/session.py", line 17, in <module>
    import botocore.session
  File "/Users/ma27/Projects/deployment-batou/.appenv/8fa9d3b1/lib/python3.12/site-packages/botocore/session.py", line 29, in <module>
    import botocore.configloader
  File "/Users/ma27/Projects/deployment-batou/.appenv/8fa9d3b1/lib/python3.12/site-packages/botocore/configloader.py", line 19, in <module>
    from botocore.compat import six
  File "/Users/ma27/Projects/deployment-batou/.appenv/8fa9d3b1/lib/python3.12/site-packages/botocore/compat.py", line 34, in <module>
    from botocore.vendored.six.moves import http_client

The requirements.txt looked like this:

batou_ext[version-select] @ https://github.com/flyingcircusio/batou_ext/archive/1d185016c0049d64c509263b5028f75ca3c2d2ac.zip#sha256=ff87879d35c0bb77fb24022f2fd8eed0c605ada48befbc04ee0648faa817cabf
batou>=2.4
boto3
docutils==0.17.1
python-dateutil==2.8.1
s3transfer==0.4.2
jmespath==0.10.0

Guess I'll do batou>=2.4,<2.5.0b3 for now.

[elikoga]

What are the preferred python versions?

The bootstrapping mechanism should only trigger when import ensurepip errors. Nixpkgs Python 3.11 includes a working ensurepip though, so the old setuptools may come from somewhere else.

[Ma27]

No preferences specified. Confirmed that prepending (# appenv-python-preference: 3.11,3.12) solves the issue and I cannot reproduce the boto3 issue in my comment above yours.

[elikoga]

Yes, we probably should create and merge https://github.com/flyingcircusio/appenv/compare/pip-freeze-all-package-regardless-of-version?expand=1

I'd love to merge https://github.com/flyingcircusio/appenv/compare/venv-create-update-deps?expand=1 but upgrade_deps was added in python 3.9 :/

[elikoga]

It looks like https://github.com/python/cpython/blob/3.11/Lib/ensurepip/init.py#L13 got us into this mess. When creating a venv, ensurepip is used to install pip (and setuptools as a pip dependency)

[elikoga]

Should be closed by #55

[Ma27]

With batou 2.5.0 I still get

Traceback (most recent call last):
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 601, in <module>
    main()
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 597, in main
    appenv.run(application_name, sys.argv[1:])
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 368, in run
    self.prepare()
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 453, in prepare
    pip(env_dir, ["check"])
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 62, in pip
    return python(path, ["-m", "pip"] + c, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 58, in python
    return cmd([os.path.join(path, "bin/python")] + c, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 54, in cmd
    raise ValueError(e.output.decode("utf-8", "replace"))
ValueError: batou 2.5.0 has requirement setuptools>=66.1.0, but you have setuptools 65.5.0.

while deploying.

Python is Python 3.11.9. Reproducible with latest appenv (just downloaded the latest version of appenv.py which should be https://github.com/flyingcircusio/appenv/commit/4692dff07b86d478309b0c3f9403ad76b522300a).

[PhilTaken]

Manually adding e.g. setuptools==66.1.0 to requirements.{txt,lock} then updating the lockfile via appenv fixes this issue

[Ma27]

Sounds like underspecified dependencies somewhere then. I don't think we should maintain a setuptools dependency in our deployments.

[PhilTaken]

I suggest we move further discussion to flyingcircusio/appenv instead since it appears the lockfile does not include setuptools by default

[elikoga]

With batou 2.5.0 I still get

Traceback (most recent call last):
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 601, in <module>
    main()
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 597, in main
    appenv.run(application_name, sys.argv[1:])
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 368, in run
    self.prepare()
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 453, in prepare
    pip(env_dir, ["check"])
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 62, in pip
    return python(path, ["-m", "pip"] + c, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 58, in python
    return cmd([os.path.join(path, "bin/python")] + c, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ma27/Projects/meine.haie.de.batou/./batou", line 54, in cmd
    raise ValueError(e.output.decode("utf-8", "replace"))
ValueError: batou 2.5.0 has requirement setuptools>=66.1.0, but you have setuptools 65.5.0.

while deploying.

Python is Python 3.11.9. Reproducible with latest appenv (just downloaded the latest version of appenv.py which should be flyingcircusio/appenv@4692dff).

After the update to the new appenv version, did the lockfile get rebuilt?

[elikoga]

Just checked, this error resolves once the lockfile is rebuilt using ./appenv update-lockfile.

We could

1. Connect the lockfile hash to the appenv version, such that new appenv versions see old lockfiles as invalid/outdated and prompt an update
2. Introduce an update command like in https://github.com/flyingcircusio/appenv/issues/39 or https://github.com/flyingcircusio/appenv/issues/1 that includes a lockfile update.

@Ma27 @PhilTaken what do you think of each of the solutions?

[Ma27]

Yes indeed.

So what I think happened was

• Updated something in a requirements.txt (with batou>=2.5.0b2)
• ./appenv update-lockfile works, but subsequent deploy fails
• I remember to update appenv, but forgot to rerun the update (which seems kinda... obvious looking back now, sorry 😅 )

Personally, I'd prefer the latter option. To me it still seems kinda awkward to just curl a new version of the new script when I need to update it.

Though, connecting the lockfile to appenv seems kinda orthogonal to the second approach: maybe we also want that since it generally seems a little more robust (however not as important to implement with (2) being done).

[PhilTaken]

I would strongly argue in favor of the first option.

It does not impact lockfile updates unless you update appenv, in which case you might as well update the requirements anyways - and should, as we can see here.

Running an appenv-specific update script is error-prone (pebcak) and tedious with no way to even tell if it has been run unless you also tack the appenv version into the lockfile at which point you've just implemented a worse version of the first option.

[Ma27]

at which point you've just implemented a worse version of the first option.

It's a good point that you'll probably need option 1 to correctly implement option 2. If that's actually the case, I fully agree.

My point is mostly motivated from the fact that I'd expect an auto-updater to be easier to use as end-user here.

Anyways, there are issues for both things, so this can be closed for good :D