[POC] Improve `importlib.metadata` usage

[mkniewallner]

Roughly a POC for now that will most likely get split into multiple PRs in the end, but thought it would be nice to have a first draft, to raise some discussion.

While working towards how to resolve https://github.com/fpgmaas/deptry/issues/827, I dug a bit into importlib.metadata that we currently use, to see if we have some ways to improve what we do.

Python 3.10 introduced packages_distributions that returns a mapping that maps Python module names to the packages that expose them, by reading from top_level.txt like we do. In Python 3.11, the method became even smarter by falling back to reading from RECORD when no top_level.txt is present, also exactly like we do.

Updates in importlib.metadata are upstreamed from importlib_metadata, so by requiring >=4.13 on Python < 3.11, we could use packages_distributions in our codebase for 2 things.

First, avoid parsing top_level.txt and RECORD ourselves, and instead simply rely on the method which already does that.

Second, and this is the most interesting part, detect more transitive dependencies, as I noticed that when building Module, we also rely on importlib.metadata and check if we detect a package <-> module mapping, but by using metadata from importlib.metadata, which is AFAIK equivalent to only checking top_level.txt, and not RECORD, meaning that some dependencies are mistakenly flagged as missing instead of transitive dependencies (like shown in https://github.com/fpgmaas/deptry/issues/827). With the changes in this PR, bs4 import in the issue is correctly reported as a transitive dependency, instead of a missing one:

$ uv run deptry .
Scanning 1 file...

foo.py:1:8: DEP003 'bs4' imported but it is a transitive dependency
Found 1 dependency issue.

Switching to packages_distributions also highlights an issue with our implementation. In Module, we assume that we can only have one package exposing a module, but in fact, it is possible for multiple packages to provide a module (for instance if using namespace packages I believe). Because of that, the report we do might be incomplete, but worse than that, this probably means that we wrongly handle some violations today, as for instance, if a module is provided by 2 packages, where one is a dev dependency and one a production one, we would need to assess the violations based on all packages we find.

As mentioned at the top of the description, I'd like to eventually split the PR into smaller pieces:

• one part (the simpler) to depend on importlib_metadata on Python < 3.11 and update Dependency to use packages_distributions, removing our own logic that reads from top_level.txt and RECORD along the way
• one or more part that updates Module to use packages_distributions, which is more complex to handle since we could now end up with multiple packages when building a module

[fpgmaas]

Great idea, I love it. Less custom logic on our side sounds like an improvement :)