Add Python 3.12 to the CI matrix

[webknjaz]

Contributor checklist

• [x] Included tests for the changes.
• [x] PR title is short, clear, and ready to be included in the user-facing changelog.

Maintainer checklist

• [x] Verified one of these labels is present: backwards incompatible, feature, enhancement, deprecation, bug, dependency, docs or skip-changelog as they determine changelog listing.
• [ ] Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

[hugovk]

3.12 is failing on the CI: https://github.com/jazzband/pip-tools/actions/runs/12346208386/job/34451449901?pr=2148

[WhyNotHugo]

Failure for piplowest is due to it referencing pkgutil.ImpImporter, which is gone is Python 3.12.

I think that we'll need to bump the minimal support version of pip.

[WhyNotHugo]

I think that failure with pipsupported might be due to to this change in pip:

freeze no longer excludes the setuptools, distribute, and wheel from the output when running on Python 3.12 or later, where they are not included in a virtual environment by default. Use --exclude if you wish to exclude any of these packages. (#4256)

[WhyNotHugo]

References to pkgutil.ImpImporter were made conditional in https://github.com/pypa/pip/commit/f1a7a6f942b869ae3428dbf55dfc1ca756c78d94, which was first included in 23.1.2, tagged 2023-04-26.

I wonder if we need to update the minimal supported pip, or if we can claim "For Python 3.12, the minimal pip is 23.1.2", while remaining compatible with older versions in Python < 3.12.

[WhyNotHugo]

See also: https://github.com/jazzband/pip-tools/issues/2104

[macro1]

pip itself only supports the most recent version. is there a reason to be careful to maximize backward compatibility?

[webknjaz]

pip itself only supports the most recent version. is there a reason to be careful to maximize backward compatibility?

I think historically, pip-tools would support the previous pip version and the current one. But now with so low activity we're not catching up with the changes on time. The README even used to have a compatibility table.

https://github.com/jazzband/pip-tools/issues/926 / https://github.com/jazzband/pip-tools/pull/1950

I didn't really want this PR to become a discussion of the support drops. Only wanted to add modern Python to the matrix. Obviously, it needs some tweaks, likely incorporating the ideas from #2105.

[macro1]

I didn't really want this PR to become a discussion of the support drops.

that's fine. we can discuss it elsewhere as well. really i meant to voice my support for dropping support for older versions over adding complexity to the pipeline

[webknjaz]

I think that for as long as a good range of Pythons is supported, it's okay to depend on newer Pip versions. But I don't have stats on why people would want to use older versions of it, honestly. This would require somebody to lead a discussion and make sure different user groups are reached, aware and heard.

[webknjaz]

References to pkgutil.ImpImporter were made conditional in pypa/pip@f1a7a6f, which was first included in 23.1.2, tagged 2023-04-26.

I wonder if we need to update the minimal supported pip, or if we can claim "For Python 3.12, the minimal pip is 23.1.2", while remaining compatible with older versions in Python < 3.12.

Post tox invocation, this is reproducible via

$ .tox/py312-piplowest/bin/python -I -m pip install               
ERROR: Exception:
Traceback (most recent call last):
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/commands/install.py", line 315, in run
    session = self.get_default_session(options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 98, in get_default_session
    self._session = self.enter_context(self._build_session(options))
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 125, in _build_session
    session = PipSession(
              ^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/network/session.py", line 343, in __init__
    self.headers["User-Agent"] = user_agent()
                                 ^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/network/session.py", line 175, in user_agent
    setuptools_dist = get_default_environment().get_distribution("setuptools")
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 180, in get_distribution
    return next(matches, None)
           ^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 177, in <genexpr>
    for distribution in self.iter_all_distributions()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/base.py", line 594, in iter_all_distributions
    for dist in self._iter_distributions():
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 168, in _iter_distributions
    for dist in finder.find_eggs(location):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 136, in find_eggs
    yield from self._find_eggs_in_dir(location)
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 103, in _find_eggs_in_dir
    from pip._vendor.pkg_resources import find_distributions
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/__main__.py", line 31, in <module>
    sys.exit(_main())
             ^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/cli/main.py", line 70, in main
    return command.main(cmd_args)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 101, in main
    return self._main(args)
           ^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
    self.handle_pip_version_check(options)
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 179, in handle_pip_version_check
    session = self._build_session(
              ^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 125, in _build_session
    session = PipSession(
              ^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/network/session.py", line 343, in __init__
    self.headers["User-Agent"] = user_agent()
                                 ^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/network/session.py", line 175, in user_agent
    setuptools_dist = get_default_environment().get_distribution("setuptools")
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 180, in get_distribution
    return next(matches, None)
           ^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 177, in <genexpr>
    for distribution in self.iter_all_distributions()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/base.py", line 594, in iter_all_distributions
    for dist in self._iter_distributions():
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 168, in _iter_distributions
    for dist in finder.find_eggs(location):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 136, in find_eggs
    yield from self._find_eggs_in_dir(location)
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py", line 103, in _find_eggs_in_dir
    from pip._vendor.pkg_resources import find_distributions
  File "~/src/github/jazzband/pip-tools/.tox/py312-piplowest/lib/python3.12/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2164, in <module>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?

[webknjaz]

@WhyNotHugo I applied the suggested update.

[sirosen]

We now have the core of this merged via #2188

I checked and I don't think there's anything more in this PR which we need to recover, so I'm closing.