Some modules missing

[takluyver]

Thanks, this is a really useful project. However, I've just tried classifying a list of loaded modules, and there are a number of standard library modules which weren't picked up. I think these are ones which aren't listed in the docs.

A few selected examples (there are others):

_hashlib
email.feedparser
encodings
encodings.utf_8
importlib._bootstrap
json.decoder
posixpath
sqlite3.dbapi2
sre_compile
typing.io  # Not a real module, but in sys.modules
xml.parsers

Maybe the answer is that the undocumented modules are out of scope for this project. I don't know of a good way to get a list including the undocumented ones. But it would be useful for my purposes to have such a list, so I'd be interested if we can figure out a way.

[takluyver]

Also a number of asyncio modules, such as asyncio.tasks.

[thebjorn]

I've collected some more modules (many, but not all of the ones you're listing) in https://github.com/thebjorn/pydeps/blob/master/pydeps/pystdlib.py

I'd love it if python-stdlib-list could be the "one true list" :-)

[takluyver]

Thanks! Did you collect them manually, or are they the result of some automated process?

[thebjorn]

The pydeps tool analyzes a module's Python import graph and IIRC I compared the found modules' __file__ attribute with sys.prefix (so sort of automated, but relying on which modules the inspected code imports).

[takluyver]

Maybe one option would be to install Python in a controlled way, and then go through the files it puts in the stdlib directory? Then there's sys.builtin_module_names to add to that. And if an automated process can do 99%, maybe the corner cases can be added by hand.

[jackmaney]

Good catch!

Yes, it seems as though it'll be necessary to go through the site-packages directory on fresh installs, as well as look at sys.builtin_module_names to help fill in those gaps. This week is a busy one, so I'll put something together this weekend.

[takluyver]

Thanks Jack :-)

[takluyver]

I've refined the tool I'm working on a bit more, and I thought that a longer list of stdlib modules it's currently missing might be useful for testing:

https://gist.github.com/takluyver/ddf61f88ad84ca458cf073f67a656019

It's split into two parts: modules which have __file__ and modules which don't (builtins, dynamically defined modules, maybe namespace packages).

[takluyver]

Any progress? Do you want me to try to have a look at this?

[takluyver]

If anyone's curious what I'm using this for, I'm trying to capture the modules loaded by an application, and then group them according to where they came from:

https://pypi.python.org/pypi/kartoffel

[woodruffw]

From a quick look, all of the modules originally listed (except for typing.io) are now included in the 3.9 listing; I'll make a new release to propagate that shortly.

Some of the others in the longer gist are still missing, though. Not sure what to do about those yet; open to ideas.

[woodruffw]

Folding this under #80.