feat: Support extras in stubtest_third_party.py

[kkirsche]

X-Ref: https://github.com/python/typeshed/pull/8457#discussion_r935592063

This pull request adds a new key to the METADATA.toml file, under the tool.stubtest key, named extras_dependencies. This key is to allow the use of extras when running stubtest in upstream tests.

For example, the current schema would change from:

version = ...
requires = ...
extra_description = ...
obsolete_since = ...
no_longer_updated = ...

[tool.stubtest]
skip = ...
apt_dependencies = ...
ignore_missing_stubs = ...

to:

version = ...
requires = ...
extra_description = ...
obsolete_since = ...
no_longer_updated = ...

[tool.stubtest]
skip = ...
apt_dependencies = ...
extras_dependencies = ["sock"]; # the list of names of extras to install.
ignore_missing_stubs = ...

The reason for this is that when a stub, such as urllib3, which provides contributor functionality based on other libraries, such as PySock, may have a complex version dependency. In those cases, rather than trying to ensure that the version specification is manually or automatically detected as updated, this bypasses the issue by ensuring that we're using the version specification defined by the upstream library, tracking it as it changes.

[AlexWaygood]

Thanks!

Do any of our stubs packages that currently have requirements-stubtest.txt files specify extra dependencies at runtime? If so, can you update the stubs for those packages so that they use this new key, instead of the requirements-stubtest.txt file, as part of this PR? That way we can see whether stubtest still works correctly on those packages with this change.

[github-actions[bot]]

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

[kkirsche]

Thanks!

Do any of our stubs packages that currently have requirements-stubtest.txt files specify extra dependencies at runtime? If so, can you update the stubs for those packages so that they use this new key, instead of the requirements-stubtest.txt file, as part of this PR? That way we can see whether stubtest still works correctly on those packages with this change.

I've updated all the stubs that currently use extras. Depending on the merge order between this and #8457, I'll merge and update whichever merge request gets merged second with any required changes

[kkirsche]

extras_dependencies is a little confusing to me, since it's not clear whether it's a list of names of extras or a list of names of dependencies. Maybe extras or extras_to_install?

I'm perfectly happy with those, but will through out a few others in case they help:

• requires_extras
• stubs_use_extras
• tests_require_extras
• stubs_require_extras

Just let me know the preference, and I can proceed with that

[kkirsche]

Updated:

1. Merged latest master to pick up the changes from #8457
2. Updated #8457 to install extras via this feature.
3. Updated extras_dependencies to extras
4. Clarification is required (either from discussion or trial and error about test failures) from the team about https://github.com/python/typeshed/pull/8467#discussion_r935911618 as lxml (as I understand it) would not be installed with the extras.

[github-actions[bot]]

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

[kkirsche]

#8439 will, if merged, also be a candidate for this via https://github.com/psf/requests/blob/main/setup.py#L125

EDIT: fix the links

[kkirsche]

Merge request updated with the extras for requests since https://github.com/python/typeshed/pull/8485 was merged.

This is using the socks extra seen here: https://github.com/psf/requests/blob/main/setup.py#L125

[github-actions[bot]]

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

[github-actions[bot]]

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

[kkirsche]

Thanks for updating, sorry for the delay here! :-)

No worries, life happens 🙂 thanks for your time