Most built-in tool lockfiles do not work with Python 3.12

[huonw]

Describe the bug

The built-in lockfiles for tools like: pytest, mypy, ..., use pip 23.0.1, which doesn't work with Python 3.12:

13:28:11.08 [INFO] Completed: Building pytest.pex from resource://pants.backend.python.subsystems/pytest.lock
13:28:11.09 [ERROR] 1 Exception encountered:

Engine traceback:
  in `test` goal

ProcessExecutionFailure: Process 'Building pytest.pex from resource://pants.backend.python.subsystems/pytest.lock' failed with exit code 1.
stdout:

stderr:
The Pip requested was pip==23.0.1 but it does not work with the interpreter selected which is CPython 3.12.0 at /Users/huon/.pyenv/versions/3.12.0/bin/python3.12. Pip 23.0.1 requires Python <3.12,>=3.7.

Reproducer:

cd $(mktemp -d)

cat > pants.toml <<EOF
[GLOBAL]
pants_version = "2.19.0rc2"

backend_packages = [
  "pants.backend.python",
]

[python]
interpreter_constraints = ["==3.12.*"]
EOF

echo 'python_tests(name="t")' > BUILD
echo 'def test_foo(): pass' > test_example.py

# BUG: fails by default with Python 3.12
pants test ::
#> The Pip requested was pip==23.0.1 but it does not work with the interpreter selected which is CPython 3.12.0 at /Users/huon/.pyenv/versions/3.12.0/bin/python3.12. Pip 23.0.1 requires Python <3.12,>=3.7.

# OKAY: works fine
pants test --python-interpreter-constraints='["==3.11.*"]' ::

Workaround: use custom lockfiles, with python_requirements/install_from_resolve.

Pants version

2.19.0rc2

(NB. 2.18.1 uses a version of pex that doesn't support Python 3.12 by default, but would likely be still affected by this if one specifies a newer pex in [pex-cli].)

OS

macOS

Additional info Discussed in https://github.com/pantsbuild/pants/pull/20310#pullrequestreview-1797974655

Details of pip version support in https://github.com/pantsbuild/pex/blob/4eb5c9aa25c6a695bf55263ab239189b720cebaf/pex/pip/version.py#L128-L233

See https://github.com/search?q=repo%3Apantsbuild%2Fpants%20pip_version%2023.0.1&type=code for 27 current lock files using pip version 23.0.1.

[huonw]

The lockfiles are mostly using 23.0.1, and that has a version range of >=3.7,<3.12: https://github.com/pantsbuild/pex/blob/4eb5c9aa25c6a695bf55263ab239189b720cebaf/pex/pip/version.py#L182-L187

Thus, I imagine it's not a problem to regenerate to a newer versions that just increase this range, while still supporting 3.7: https://github.com/pantsbuild/pex/blob/4eb5c9aa25c6a695bf55263ab239189b720cebaf/pex/pip/version.py#L210-L233

That said, pip suggests that pants' default pip version (23.1.2) doesn't work with python 3.12 (this seems potentially incorrect? https://github.com/pantsbuild/pex/pull/2314), so we might need to leap to an even newer pip and/or bump the default pip version too.