This is a meta issue tracking all the things that need to be cleaned up in the import system. From removing long deprecated APIs to migrating internal access to __spec__, we'll have several task lists for each related set of clean ups.

@brettcannon @ericsnowcurrently for visibility.

Removing deprecated APIs

Over in What's New for Python 3.11 there's a long list of things that have been deprecated long enough that they can be removed in Python 3.12. Here's that task list for tracking purposes. Watch for linked PRs (maybe one big one or many small ones -- we'll see!).

@brettcannon @ericsnowcurrently for visibility.

[ ] importlib.find_loader()
[x] importlib.util.module_for_loader()
[ ] importlib.util.set_loader_wrapper()
[ ] importlib.util.set_package_wrapper()
[x] importlib.abc.Loader.module_repr() (3.12: #97876)
[ ] importlib.abc.MetaPathFinder.find_module()
[ ] importlib.abc.MetaPathFinder.find_module()
[ ] importlib.abc.PathEntryFinder.find_loader()
[ ] importlib.abc.PathEntryFinder.find_module()
[ ] importlib.machinery.BuiltinImporter.find_module()
[x] importlib.machinery.BuiltinLoader.module_repr() (3.12: #97876)
[ ] importlib.machinery.FileFinder.find_loader()
[ ] importlib.machinery.FileFinder.find_module()
[ ] importlib.machinery.FrozenImporter.find_module()
[x] importlib.machinery.FrozenLoader.module_repr() (3.12: #97876)
[ ] importlib.machinery.PathFinder.find_module()
[ ] importlib.machinery.WindowsRegistryFinder.find_module()
[ ] Remove pkgutil.ImpImporter (3.12: already deprecated)

Update docs

[ ] Make sure examples and details from PEP 302 are covered in the importlib docs (/cc @Yhg1s )
[x] Remove https://docs.python.org/3/reference/import.html#open-issues (PR)
[ ] It would be really nice to have a diagram.
[ ] (import_machinery.rst) how about a section devoted just to the attributes of modules and packages, perhaps expanding upon or supplanting the related entries in the data model reference page?
[ ] runpy, pkgutil, et al in the library manual should all get “See Also” links at the top pointing to the new import system section.
[ ] Add more explanation regarding the different ways in which __main__ is initialized?
[ ] Add more info on __main__ quirks/pitfalls (i.e. copy from PEP 395).

Missing deprecations

Here's a list of related APIs that have not yet been deprecated, but should be. We'll deprecate them in 3.12 and remove them in 3.14.

[ ] pkgutil.find_loader()
[ ] pkgutil.get_loader() (or it needs to be reimplemented to not use find_loader())

PR: gh-98520

Oct 04 '22 22:10 warsaw

I've removed importlib.utils.set_loader, importlib.utils.set_package, and importlib.utils.module_for_loader from importlib.utils in https://github.com/python/cpython/pull/97898 🎉

Oct 05 '22 08:10 sobolevn

I've created a PR: https://github.com/python/cpython/pull/98520 to complete these two tasks:

[ ] pkgutil.find_loader()

[ ] pkgutil.get_loader() (or it needs to be reimplemented to not use find_loader())

Oct 21 '22 13:10 sobolevn

FYI https://github.com/python/cpython/issues/64138 is a tracking issue for pkgutil all up.

Oct 21 '22 18:10 brettcannon

From the meta checklist:

Deprecate pkgutil.find_loader()

I kind of think we're okay here, at least for 3.12. This uses importlib.util.find_spec() so I feel like it's not so urgent to deprecate and actually probably has some utility.

Deprecate pkgutil.get_loader()

Similarly, I'm not sure this even needs to be deprecated. It is implemented in terms of the above, and thus find_spec(), and it has some potentially useful wrapper semantics.

I'm going to mark both issues as complete and not deprecate it in 3.12.

Apr 29 '23 23:04 warsaw

@brettcannon @ericsnowcurrently - I think this PR is ready for review. I feel good that we've implemented all the removals we need for 3.12. Like the other issues, we can address the doc updates after beta 1.

Apr 29 '23 23:04 warsaw

The removal branch has been merged, but I'm leaving this meta issue open for now so we can address the documentation checklist during the beta cycle.

May 03 '23 11:05 warsaw

Thanks again for tackling this, @warsaw!

May 03 '23 14:05 ericsnowcurrently

[For the imp module, I created issues for most of the 71 projects in the PyPI top 5k (listed at python/cpython#98573 (comment)), where I could find their tracker. And some already had issues.

I've checked off projects which had already fixed but not yet released, or have fixed since I created the issue, or where the project is unmaintained or otherwise abandoned and a fix is unlikely in any case.

Currently 24/71 (34%) are checked off. In the issue I pointed to the 3.11 imp docs which have suggestions of how to upgrade each functions/constants, and most seemed fine with that so far, except for one noting imp.load_source has no suggestions.

[x] 1. ansiwrap - not updated since 2019, already reported: https://github.com/jonathaneunice/ansiwrap/issues/17 and has 3.11 test failures: https://github.com/jonathaneunice/ansiwrap/issues/18
[x] 2. asttokens - only in a file used as data for a test - reported: https://github.com/gristlabs/asttokens/issues/108
[x] 3. audioread - already replaced: https://github.com/beetbox/audioread/pull/128
[x] 4. auth - imported but not used, merged PR: https://github.com/ourway/auth/pull/9
[x] 5. autopep8 - used in tests, reported: https://github.com/hhatto/autopep8/issues/683
[x] 6. boto - "This package is no longer maintained and has been replaced by Boto3"
[x] 7. cement - already reported: https://github.com/datafolklabs/cement/issues/386
[x] 8. coremltools - reported: https://github.com/apple/coremltools/issues/1846
[x] 9. crochet - reported: https://github.com/itamarst/crochet/issues/145
[x] 10. Cython - looks like compat aleady added
[x] 11. databricks-cli - reported: https://github.com/databricks/databricks-cli/issues/631
[x] 12. datalab - project is deprecated, repo is archived
[x] 13. diff-match-patch - used in tests, reported: https://github.com/diff-match-patch-python/diff-match-patch/issues/5
[x] 14. distribute - not updated since 2013, "This package is a simple compatibility layer that installs Setuptools 0.7+."
[x] 15. django_compressor - https://github.com/django-compressor/django-compressor/issues/1185
[x] 16. django-compat - not updated in 6 years but reported: https://github.com/arteria/django-compat/issues/67
[x] 17. django-configurations - reported: https://github.com/jazzband/django-configurations/issues/358
[x] 18. django-heroku - not updated in 5 years and repo archived
[x] 19. dominate - reported https://github.com/Knio/dominate/issues/172
[x] 20. flasgger - reported https://github.com/flasgger/flasgger/issues/563
[ ] 21. Flask-AppBuilder - reported https://github.com/dpgaspar/Flask-AppBuilder/issues/2027
[ ] 22. future - already reported: https://github.com/PythonCharmers/python-future/issues/246
[x] 23. gevent - already has compat (via _imp)
[x] 24. glom - reported https://github.com/mahmoud/glom/issues/261
[x] 25. gsutil - reported https://github.com/GoogleCloudPlatform/gsutil/issues/1695, with PR: https://github.com/mtth/hdfs/pull/192
[x] 26. hdfs - already reported: https://github.com/mtth/hdfs/issues/191
[x] 27. hyper - project archived
[x] 28. IMAPClient - in tests, reported https://github.com/mjs/imapclient/issues/504
[x] 29. invoke - fixed and released: https://github.com/pyinvoke/invoke/pull/919
[ ] 30. j2cli - reported https://github.com/kolypto/j2cli/issues/80
[x] 31. joblib - test file removed
[x] 32. Kivy - reported https://github.com/kivy/kivy/issues/8225
[ ] 33. logilab-common
[x] 34. lxml - reported https://bugs.launchpad.net/lxml/+bug/2018137
[x] 35. markdown2 in tests, reported https://github.com/trentm/python-markdown2/issues/510
[ ] 36. mercurial
[x] 37. metaflow - reported https://github.com/Netflix/metaflow/issues/1380
[x] 38. ml_collections - reported https://github.com/google/ml_collections/issues/24
[x] 39. mysql-connector - no release since 2017, likely unmaintained
[x] 40. mysql-connector-python-rf - deprecated
[x] 41. nose - unmaintained
[x] 42. numba - in tests, reported https://github.com/numba/numba/issues/8933
[x] 43. oletools - reported https://github.com/decalage2/oletools/issues/813
[x] 44. os_sys - reported https://github.com/Matthijs990/os-sys-github/issues/4 but repo since deleted
[x] 45. Paste - reported https://github.com/cdent/paste/issues/75
[x] 46. pathtools - not released in 12 years
[ ] 47. pbr - reported: https://bugs.launchpad.net/pbr/+bug/2018770
[x] 48. pep562 - reported https://github.com/facelessuser/pep562/issues/13, but no longer receiving active support
[x] 49. pex - reported https://github.com/pantsbuild/pex/issues/2137
[x] 50. ptvsd - project is deprecated, repo is archived
[ ] 51. pydevd-pycharm - reported: https://github.com/fabioz/PyDev.Debugger/issues/251
[ ] 52. pydoop - in tests, reported https://github.com/crs4/pydoop/issues/371
[ ] 53. pyexcel - in tests, reported https://github.com/pyexcel/pyexcel/issues/274
[x] 54. pyinstaller - reported https://github.com/pyinstaller/pyinstaller/issues/7601
[x] 55. pynose - reported https://github.com/mdmintz/pynose/issues/1
[ ] 56. pyrepl - https://github.com/pypy/pyrepl
[x] 57. pysaml2 - reported https://github.com/IdentityPython/pysaml2/issues/904
[ ] 58. pytest-shutil - reported https://github.com/man-group/pytest-plugins/issues/216
[x] 59. PyXB - project only works up to 3.7 https://github.com/pabigot/pyxb/issues/100#issuecomment-1526932482 and repo archived
[ ] 60. reportlab
[x] 61. snapshottest - reported https://github.com/syrusakbary/snapshottest/issues/166 but unmaintained: https://github.com/syrusakbary/snapshottest/issues/174
[ ] 62. social-auth-core - reported https://github.com/python-social-auth/social-core/issues/789
[x] 63. sox - reported https://github.com/rabitt/pysox/issues/158
[ ] 64. sqlalchemy-migrate
[x] 65. stone - reported https://github.com/dropbox/stone/issues/295
[x] 66. strip-hints - reported https://github.com/abarker/strip-hints/issues/19
[x] 67. Theano - no longer maintained https://github.com/Theano/Theano
[x] 68. Twisted - fixed pending release https://github.com/twisted/twisted/pull/11712
[x] 69. twofish - already reported in 2016 but no updates since 2013 https://github.com/keybase/python-twofish/issues/4
[ ] 70. uwsgi - reported https://github.com/unbit/uwsgi/issues/2534
[x] 71. workflow - reported https://github.com/inveniosoftware-contrib/workflow/issues/68 but unmaintained: https://github.com/inveniosoftware-contrib/workflow/issues/66

Edit 2024-06-07: 56/71 (79%)

May 03 '23 15:05 hugovk

Due to nose being unmaintained (number 41 in your list), that's why there's pynose now (number 55 in your list). It's a complete drop-in replacement for all occurrences. (Let me know if that isn't the case.)

May 04 '23 00:05 mdmintz

Edit 2023-11-05: 36/71 (52%)

We're now over 50% of those in the checklist.

Nov 05 '23 20:11 hugovk

Edit 2024-06-07: 55/71 (78%)

We're now at 78% (fixed or unmaintained/abandoned).

Jun 07 '24 09:06 hugovk

cpython
cpython copied to clipboard

Meta issue for cleaning up import system cruft

Removing deprecated APIs

Update docs

Missing deprecations

cpython cpython copied to clipboard

Meta issue for cleaning up import system cruft

Removing deprecated APIs

Update docs

Missing deprecations

cpython
cpython copied to clipboard