typeshed
typeshed copied to clipboard
python
Add `distutils` as a top-level package included with `types-setuptools`
~~This breaks stubtest_third_party.py when you run it on Python <3.12 (as we do in CI), unfortunately, as it gets very confused at there being both a stdlib/distutils package and a stubs/setuptools/distutils package. Not sure how to fix that...~~
Closes #10255
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
I got stubtest working, with a few hacks. If we're happy with the hacks, I can ask the pyright maintainers to look at why the test cases I added might be failing.
Or we could just delete the test cases for now, and try to figure out later why it isn't working for pyright. I don't think this PR would cause any regressions for pyright users; it just wouldn't offer the same benefits as it's offering for mypy users
I would prefer if we copy the distutils stubs into setuptools. From what I understand, they are different from stdlib's distutils at this point anyway, and may evolve further. I'd even argue that copied stubs might be easier to maintain: We probably won't need to update stdlib's distutils much, but might need to update setuptools's from time to time.
I would prefer if we copy the distutils stubs into setuptools. From what I understand, they are different from stdlib's distutils at this point anyway, and may evolve further. I'd even argue that copied stubs might be easier to maintain: We probably won't need to update stdlib's distutils much, but might need to update setuptools's from time to time.
We've already done that. We copied them into setuptools/_distutils some time ago. This PR attempts to reflect the fact that setuptools essentially re-exports the entirety of setuptools._distutils as a standalone distutils package.
At runtime, setuptools._distutils becomes distutils via a crazy importlib hack, but we can't do that in the stubs.
Does it still use an import hack? At least the freshly installed version of setuptools (68.2.2) vendors distutils and does seem to use it:
Python 3.11.4 (main, Jun 9 2023, 07:59:55) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools._distutils as d
>>> d.__file__
'/home/srittau/foo/lib/python3.11/site-packages/setuptools/_distutils/__init__.py'
>>> import setuptools._distutils.log as l
>>> l.__file__
'/home/srittau/foo/lib/python3.11/site-packages/setuptools/_distutils/log.py'
setuptools._distutils does not use an import hack, and never has done, but the plain distutils package that's installed into your environment as part of pip install setuptools still does.
But setuptools.distutils is equivalent to setuptools._distutils, at least in my tests:
System Python:
Python 3.11.4 (main, Jun 9 2023, 07:59:55) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from setuptools import distutils as d
>>> d.__file__
'/usr/lib/python3/dist-packages/setuptools/_distutils/__init__.py'
Virtualenv with locally updated setuptools:
Python 3.11.4 (main, Jun 9 2023, 07:59:55) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from setuptools import distutils as d
>>> d.__file__
'/home/srittau/foo/lib/python3.11/site-packages/setuptools/_distutils/__init__.py'
Not from setuptools import distutils, just plain import distutils.
pip installing setuptools into your environment also installs a distutils package into your environment that is standalone to setuptools.
And yes, it gets you the same thing as setuptools._distutils. That's what this PR is trying to reflect! :)
@srittau, this is the behaviour I'm trying to reflect in this PR:
(fresh-venv) C:\Users\alexw\coding>python -VV
Python 3.12.0 (tags/v3.12.0:0fb18b0, Oct 2 2023, 13:03:39) [MSC v.1935 64 bit (AMD64)]
(fresh-venv) C:\Users\alexw\coding>python -m pip install --upgrade pip
Requirement already satisfied: pip in c:\users\alexw\coding\fresh-venv\lib\site-packages (23.2.1)
Collecting pip
Obtaining dependency information for pip from https://files.pythonhosted.org/packages/47/6a/453160888fab7c6a432a6e25f8afe6256d0d9f2cbd25971021da6491d899/pip-23.3.1-py3-none-any.whl.metadata
Using cached pip-23.3.1-py3-none-any.whl.metadata (3.5 kB)
Using cached pip-23.3.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 23.2.1
Uninstalling pip-23.2.1:
Successfully uninstalled pip-23.2.1
Successfully installed pip-23.3.1
(fresh-venv) C:\Users\alexw\coding>python -c "import setuptools"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'setuptools'
(fresh-venv) C:\Users\alexw\coding>python -c "import distutils"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'distutils'
(fresh-venv) C:\Users\alexw\coding>python -m pip install setuptools
Collecting setuptools
Downloading setuptools-68.2.2-py3-none-any.whl.metadata (6.3 kB)
Downloading setuptools-68.2.2-py3-none-any.whl (807 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 807.9/807.9 kB 4.3 MB/s eta 0:00:00
Installing collected packages: setuptools
Successfully installed setuptools-68.2.2
(fresh-venv) C:\Users\alexw\coding>python -c "import setuptools, distutils"
(fresh-venv) C:\Users\alexw\coding>python
Python 3.12.0 (tags/v3.12.0:0fb18b0, Oct 2 2023, 13:03:39) [MSC v.1935 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils
>>> distutils
<module 'distutils' (C:\Users\alexw\coding\fresh-venv\Lib\site-packages\setuptools\_distutils\__init__.py)>
In pywin32 I just hit error: Cannot find implementation or library stub for module named "distutils" [import-not-found] with mypy that only happens in Python 3.12, because it's using setuptools but still needs distutils functionality to build executable files.
https://github.com/mhammond/pywin32/actions/runs/8305626045/job/22732748471?pr=2211#step:5:11
I think this PR would resolve that issue as it would add distutils-stubs when installing types-setuptools ?
@erictraut Is there some special-casing in pyright that prevents detecting a would-be distutils-stubs from the typeshed path when checking against Python 3.12 ?
@Avasam, no, pyright has no special-case knowledge of distutils or its stub file. However, the VERSIONS file in typeshed says that the stub is unsupported on Python 3.12: distutils: 3.0-3.11. That's probably the cause.
@erictraut, the VERSIONS file only gives type checkers information regarding the stdlib stubs (this is why the file is in our stdlib/ directory). I don't think pyright should be inferring from that file that the third-party distutils stubs being added here are not available on py312+.
This situation arises because the distutils package is no longer part of the stdlib on py312+, and is now instead provided by the third-party setuptools distribution.
@AlexWaygood, that's right. On python 3.11 and earlier, pyright is resolving the import to the stdlib stub. On python 3.12 and newer, the VERSIONS file indicates that it should no longer use the stdlib stub for resolution. Since there is no other distutils library or stub package present, pyright reports that it cannot be resolved in this case. I think that's the correct behavior, no?
I'm not sure what you mean by "provided by the third-party setuptools distribution. I've tried installing the latest setuptools from pypi, and I don't see any distutils package installed along with it. I don't see where pyright would find a distutils.pyi or distutils/__init__.pyi file in this case.
I'm not sure what you mean by "provided by the third-party
setuptoolsdistribution. I've tried installing the latestsetuptoolsfrom pypi, and I don't see anydistutilspackage installed along with it. I don't see where pyright would find adistutils.pyiordistutils/__init__.pyifile in this case.
@erictraut, I mean the behaviour I described in this comment here: https://github.com/python/typeshed/pull/10948#issuecomment-1789076544. setuptools includes a copy of distutils in this directory here: https://github.com/pypa/setuptools/tree/main/setuptools/_distutils. This is exposed for users (via some importlib hacks that we don't need to worry about here) as a top-level package that is installed and available when you do pip install setuptools. This PR adds an equivalent stubs package to typeshed's stubs/setuptools directory, in an attempt to get type checkers to understand that import distutils still works on Python 3.12+ if you've pip installed setuptools. However, the failing tests in this PR indicate that pyright still does not understand import distutils to work on Python 3.12 if setuptools has been installed, even with the third-party distutils stubs I'm adding as part of this PR. I'm unclear as to why that might be.
I'm not sure what you mean by "provided by the third-party
setuptoolsdistribution. I've tried installing the latestsetuptoolsfrom pypi, and I don't see anydistutilspackage installed along with it. I don't see where pyright would find adistutils.pyiordistutils/__init__.pyifile in this case.
Setuptools adds their own vendored distsutils to the path. Of course this means it can't be analyzed statically and pyright users have to turn off reportMissingModuleSource, but that's where stubs come in to fill in the gap.
Huh, I see that this seems to be fixed on the latest version of pyright; all CI is now passing on this PR. I wonder what changed, and when.
Sorry for the noise, @erictraut -- thanks for your time, as ever!
I think the only thing still blocking this PR is the question of whether we're happy with the hacks I'm introducing to stubtest_third_party.py in order to get stubtest to still pass? Any thoughts on that from any other maintainers?
Huh, seems like the stubtest issue has also been magically fixed by one of our mypy upgrades since I originally filed this PR in October? Or maybe the fact that we now run third-party stubtest on py311 rather than py310? No idea. Anyway, looks like we don't even need the stubtest hacks anymore!