arvidn
Pypi builds are missing type stubs
Hi,
It seems that RC branch does not include the type hints, is it possible to include those in pypi?
I see that there were discussions on this before https://github.com/arvidn/libtorrent/issues/6730
cc: @AllSeeingEyeTolledEweSew
@qstokkink You interested in submitting a PR? (Since you have experience with "type stubs" -> #7691)
https://github.com/AllSeeingEyeTolledEweSew/libtorrent/pulls/AllSeeingEyeTolledEweSew might provide some inspiration.
cc: @AllSeeingEyeTolledEweSew
Hasn't appeared to be active for a long time.
This is just a matter of hanging the master typing-related commits into the RC branch: relatively easy[^1]. However, I assume the types were parked on master because Arvid didn't want them in an RC yet, for some reason. Of course, he may have also forgotten the types exist, I have no way of knowing. Probably best to ask him instead of me.
[^1]: The typing is a bit out of date. A little bit of love is needed to get it up to date.
one challenge with type stubs is to make sure they remain correct and aligned with the actual bindings. having a CI check for that (something like stubtest) would be important. If there's a CI test for that, I don't have any opinion about including it in RC_2_0 or not.
If that's the case, I can have a look. I don't mind scouting out some options.
If successful, I can create a PR. If not, I'll report back with how/why I failed.
EDIT: Options so far:
-
stubtest► ✅ passed, detected missing stubs and inconsistencies. -
pyright,--verifytypes❌ Failed to detect that
libtorrent.load_torrent_*bindings are not present in__init__.pyi.libtorrent Symbols used in public interface: libtorrent.add_magnet_uri /home/quinten/Documents/lttypechecker/venv/lib/python3.12/site-packages/libtorrent/__init__.pyi:66:5 - error: Type of parameter "_session" is partially unknown Parameter type is "session" libtorrent.session.add_torrent /home/quinten/Documents/lttypechecker/venv/lib/python3.12/site-packages/libtorrent/__init__.pyi:1337:9 - error: Type of parameter "resume_data" is partially unknown Parameter type is "bytes | int | List[Unknown] | Dict[bytes, Any] | Tuple[int, ...] | None" Type argument 1 for class "list" has unknown type libtorrent.session.dht_put_immutable_item /home/quinten/Documents/lttypechecker/venv/lib/python3.12/site-packages/libtorrent/__init__.pyi:1356:9 - error: Type of parameter "_entry" is partially unknown Parameter type is "bytes | int | List[Unknown] | Dict[bytes, Any] | Tuple[int, ...]" Type argument 1 for class "list" has unknown type libtorrent.session.load_state /home/quinten/Documents/lttypechecker/venv/lib/python3.12/site-packages/libtorrent/__init__.pyi:1384:9 - error: Type of parameter "entry" is partially unknown Parameter type is "bytes | int | List[Unknown] | Dict[bytes, Any] | Tuple[int, ...]" Type argument 1 for class "list" has unknown type libtorrent.bdecode /home/quinten/Documents/lttypechecker/venv/lib/python3.12/site-packages/libtorrent/__init__.pyi:69:5 - error: Return type is partially unknown Return type is "bytes | int | List[Unknown] | Dict[bytes, Any] | Tuple[int, ...]" Type argument 1 for class "list" has unknown type libtorrent.bencode /home/quinten/Documents/lttypechecker/venv/lib/python3.12/site-packages/libtorrent/__init__.pyi:71:5 - error: Type of parameter "_bdecoded" is partially unknown Parameter type is "bytes | int | List[Unknown] | Dict[bytes, Any] | Tuple[int, ...]" Type argument 1 for class "list" has unknown type libtorrent.dht_immutable_item_alert.item /home/quinten/Documents/lttypechecker/venv/lib/python3.12/site-packages/libtorrent/__init__.pyi:508:5 - error: Type argument 1 for class "list" has unknown type libtorrent.dht_mutable_item_alert.item /home/quinten/Documents/lttypechecker/venv/lib/python3.12/site-packages/libtorrent/__init__.pyi:542:5 - error: Type argument 1 for class "list" has unknown type Symbols exported by "libtorrent": 1556 With known type: 1545 With ambiguous type: 0 With unknown type: 11 Other symbols referenced but not exported by "libtorrent": 195 With known type: 195 With ambiguous type: 0 With unknown type: 0 Symbols without documentation: Functions without docstring: 385 Functions without default param: 38 Classes without docstring: 179 Type completeness score: 99.3% Completed in 0.641sec
flake8-pyi, https://github.com/PyCQA/flake8-pyi ► ✅ℹ️ failed the assignment. However, detected use of deprecated typing constructions.
Current best approach: combine python -m mypy.stubtest libtorrent with flake8 __init__.pyi
@arvidn What's your policy on picking commits from master into an RC_* branch? cherry-pick/rebase/other?
For reference, I think these are all the necessary commits, starting from commit aaa5691 as the first commit:
pick c6b03bac9 Add type stubs and tests for sha256_hash
pick 6247153cd Add binding for announce_flags_t
pick 421a00af4 Add binding to fix session.add_port_mapping
pick 17259c7db Remove binding for session_params.ext_state
pick 8b386b678 Fix bindings for members of picker_log_alert
pick 0ec312e34 Fix deprecated session.add_torrent form to prevent crash
pick 3a35e1583 Fix dht_[im]mutable_item_alert.item to just be the item
pick 9ad9d82fe Add session.session_state()
pick 4812bfbe3 update python type stubs
pick 258fd0f87 fix python tests when disabling deprecated functions
pick 7c698d826 fix wheel types not being found
pick 618992875 Moved type stubs into libtorrent directory
pick 1d644d19c update name, listen_succeeded_alert_socket_type_t
Given that these commits build on the test.py -> tests/ directory split. I think it would be best to rewrite these commits using cherry-pick, to avoid pulling in the plethora of related files.
For now, I'm proceeding with cherry-pick. The type-fixing + CI commit I build on top of that should be applicable either way.
If anyone is looking for an update before the weekend: when I said "relatively easy" before, what I should've said is "Repetitive Strain Injury ". Current status: commits cherry-picked, types modernized, types fixed, now double-checking work.
Alright guys, I finished: https://github.com/qstokkink/libtorrent/actions/runs/16753752474
At this point:
- All types are consistent, enforced with CI.
- All types are modernized, enforced with CI.
- There are no missing types, enforced with CI.
That said, there is one thing that the CI does not do and even cannot do:
- Actually call the code with the given function signatures.
So, practically, this means that if the libtorrent wheel exposes a function f(a: some_type), a function f() has to exist in the type stubs. However, the arguments to f() in the type stubs can be complete nonsense. The "only" way[^1] to catch this is by calling the method (for example, in the unit tests) and checking if the types match up at runtime.
If anyone has some spare time, it would nice if you could install one of the built wheels (that I linked to in the GH Actions run above) and let me know if you spot any function signature mismatches. I opened up a PR for reviews, if you do spot any inconsistencies.
[^1]: Boost.Python also generates __doc__ strings for functions at runtime. So, I guess technically you can read that without necessarily calling anything.