[Hinty] Add type hints to Scapy

[gpotter2]

Project "Hinty" aims at adding Type hints to Scapy. It will help discover bugs, improve the API, and make Scapy up-to-date with the high standards of Python libraries.

Implementation

We use mypy to ensure automatic testing of the work that has already been completed. PRs that fall under project Hinty will process one (or a few) files and register them into the checks. The file .config/mypy/mypy_enabled.txt lists the files in which mypy checks are enabled. This process has been added by secdev/scapy#2162

Because we support Python 2.7, the format of the Type hints should be the following: https://mypy.readthedocs.io/en/latest/cheat_sheet.html

Guide on how to contribute

0. :cd: Make sure you've "git clonned" the latest version of scapy and are working from within it !

1. :pencil2: Pick a file you want to work on. If you're new on scapy, pick a file from within the layers/ or contrib/ folder that is small in size and complexity: at this point in time, the remaining layers in Scapy's core are often hard to type.

2. :pencil2: Add that file to .config/mypy/mypy_enabled.txt. This will enable it in our test suite.

3. ➡️ pip install pyannotate cryptography tox mock (install tox, pyannotate and cryptography)

4. ➡️ Run the tests for your file: tox --sitepackages -- -x -K tcpdump -K manufdb -K wireshark -K tshark -K brotli -K zstd -K ci_only -K not_pyannotate -N You shall specify a -e py[38]-[linux]_non_root (fill the brackets with your OS infos) option before the --sitepackages option if you have multiple versions of python installed. To see all available tox tags (the whole py...non_root string), type tox -l. The -x option tells our test suite to use pyannotate. This will create test/pyannotate_results

   • ⚠️ ✏️ In many cases, pyannotate will glitch out (https://github.com/dropbox/pyannotate/issues/92). You need to open pyannotate_results with your favorite editor and replace all occurrences of <locals> (with the <>) by blanks (or anything really). For instance on vim: :%s/<locals>//g

5. ➡️ pyannotate --type-info test/pyannotate_results -w [the file.py you are working on]

6. ✏️ The file has been automatically processed. Now, edit it to fix the mistakes, and check your work with tox -e mypy. This might take time

   • ℹ️ Remember that you can use cast(type, obj) from typing to help mypy without affecting the code.
   • ⚠️ Import any types you might need from scapy.compat instead of typing. It provides a fallback if typing isn't installed (python < 3.5). I know it's annoying.
   • ℹ️ If a class extends Field directly (very few do) it will require to be passed the internal and machine representation of the field. See this if you don't know what I'm talking about. For instance: class subField(Field[float, int]): is a field where i=float and m=int. In most cases, it should be [int, int] (a normal integer field).

7. ⚠️ If you get error: No library stub file for module ..., add an exception for it in .config/mypy/mypy.ini (follow the existing format)

8. ✅ Check that the files still pass PEP8: tox -e flake8

9. 🥇 When everything passes, submit a PR. :smile:

Active hinty related issues

• Discuss strict equality in Mypy https://github.com/secdev/scapy/issues/2823

Documentation for advanced typing (that I always lose the links for..)

• Generics
• Methods overloading

Hinty status

The numbers correspond to the amount of lines per files processed MyPy Support: 23.96%

• [core]: 100.00%
• arch: 43.33%
• layers: 2.87%
• contrib: 13.86%
• libs: 0.00%
• asn1: 100.00%
• tools: 0.00%
• modules: 0.00%

[guedou]

It this compatible with type hints? https://docs.python.org/3/library/typing.html

I had a look at static typing for Scapy, and it could helps us catch some bugs and improve the API consistency.

[gpotter2]

It would be pretty amazing to get static typing, but it's not supported below Python 3.5.

Python 3.4 already had its EOL in March, so the only thing that's preventing us from doing that is the 2.7 support.

[guedou]

There is a type hints syntax that works with Python2: https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code

It works fine with our code base =)

[gpotter2]

Because we have great tests, it's really easy to generate automatic type hinting wigh pyannotate based on the values taken during execution.

I've setup the utilitary classes in https://github.com/secdev/scapy/pull/2162

[guedou]

Awesome!

I tried pyannotate but I did not find time to do more than a quick PoC. I remind that mypy outputs a lot of errors that need to be fixed. I had to fix things (like explicitly adding members to Conf). Also, flake8 complains about unused imports and I did not think of a better way to fix this than using noqa.

Do you know if we can exclude files (i.e. six.py & winpcapy.py) from a mypy run?

I propose “hinty” for the project name.

[JaninePrukop]

I'm reading through your documentation and formatting, but I would like to contribute to this. This is my first open source project, so I'd appreciate any patience/understanding, but this seems like a very doable thing.

[gpotter2]

Hi @JaninePrukop, thanks for your interest in Scapy !

The guide should be quite straightforward, though brand new (Feel free to ask any questions you have).

[nnrepos]

I'm starting work on ansmachine.py

[nnrepos]

@gpotter2

3. This will create test/pyannotate_results

This is true only if the script is run from the test folder. Is that your intention?

[nnrepos]

when running pyannotate, I get:

pyannotate --type-info ./pyannotate_results -w scapy/ansmachine.py Traceback (most recent call last): File "C:\Python27\Lib\runpy.py", line 174, in run_module_as_main "main", fname, loader, pkg_name) File "C:\Python27\Lib\runpy.py", line 72, in run_code exec code in run_globals File "C:\Users\niviman\code\python\scapy\venv\Scripts\pyannotate.exe_main.py", line 7, in File "c:\users\niviman\code\python\scapy\venv\lib\site-packages\pyannotate_tools\annotations_main.py", line 124, in main only_simple=args.only_simple) File "c:\users\niviman\code\python\scapy\venv\lib\site-packages\pyannotate_tools\annotations\main.py", line 60, in generate_annotations_json_string signature = unify_type_comments(item.type_comments) File "c:\users\niviman\code\python\scapy\venv\lib\site-packages\pyannotate_tools\annotations\main.py", line 27, in unify_type_comments arg_types, return_type = infer_annotation(type_comments) File "c:\users\niviman\code\python\scapy\venv\lib\site-packages\pyannotate_tools\annotations\infer.py", line 45, in infer_annotation arg_types, return_type = parse_type_comment(comment) File "c:\users\niviman\code\python\scapy\venv\lib\site-packages\pyannotate_tools\annotations\parse.py", line 216, in parse_type_comment return Parser(comment).parse() File "c:\users\niviman\code\python\scapy\venv\lib\site-packages\pyannotate_tools\annotations\parse.py", line 225, in init self.tokens = tokenize(comment) File "c:\users\niviman\code\python\scapy\venv\lib\site-packages\pyannotate_tools\annotations\parse.py", line 193, in tokenize raise ParseError(original) pyannotate_tools.annotations.parse.ParseError: Invalid type comment: (str) -> ctypes:CDLL.init.._FuncPtr

it seems like it ignores the -w flag and instead scans python38\lib\ctypes. I couldn't install pyannotate on python27 if that's a possible solution.

please help

[gpotter2]

I don't know much about pyannotate related issues. Is your path correct ? Retry using a more up to date version ? Using Python 3?

[akshaycbor]

I'm starting work on scapy/utils.py. One thing I noticed is that when running tox -e mypy, I get some errors on main.py and http2.py as well. Is this expected?

[guedou]

@akshaycbor these errors are related to the activation of checks concerning scapy/utils.py.

[akshaycbor]

Ok, so I tried adding plist.py(after removing utils.py) to the mypy_enabled file, and I get the same errors on the main.py and http2.py file

scapy/main.py:159: error: Module has no attribute "iteritems" scapy/main.py:395: error: Module has no attribute "iteritems" scapy/plist.py:28: error: Cannot find implementation or library stub for module named 'scapy.modules.six.moves' scapy/plist.py:28: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports scapy/plist.py:302: error: Module has no attribute "iteritems" scapy/plist.py:305: error: Module has no attribute "iteritems" scapy/contrib/http2.py:110: error: Module has no attribute "integer_types" scapy/contrib/http2.py:202: error: Module has no attribute "integer_types" scapy/contrib/http2.py:220: error: Module has no attribute "integer_types" scapy/contrib/http2.py:346: error: Module has no attribute "integer_types" scapy/contrib/http2.py:349: error: Module has no attribute "integer_types" scapy/contrib/http2.py:353: error: Module has no attribute "integer_types" scapy/contrib/http2.py:2071: error: Module has no attribute "integer_types" scapy/contrib/http2.py:2429: error: Module has no attribute "iteritems" scapy/contrib/http2.py:2448: error: Module has no attribute "iteritems"

All of these are functions in the six.py module. And I can't figure out how to fix them.

[gpotter2]

I think the six.moves module is considered as separate. Should be fixed in https://github.com/secdev/scapy/pull/2357.

[fennellkyle]

I'd like to help but am running into a couple issues. UTscapy is in scapy/scapy/tools and from there it cannot import scapy.

Traceback (most recent call last):
  File "UTscapy.py", line 27, in <module>
    from scapy.consts import WINDOWS
ModuleNotFoundError: No module named 'scapy'

If I move it to the project root it runs fine now, but pyannotate_results will be in the project root and not in test/ as the 4th step suggests: pyannotate --type-info test/pyannotate_results -w [the file.py you are working on]. Not a show stopper, I just change the path. But then pyannotate errors out: pyannotate_tools.annotations.parse.ParseError: Invalid type comment: (type, *type) -> scapy.modules.six:with_metaclass.<locals>.metaclass

Can you offer guidance? Would love to contribute as much as I can.

[gpotter2]

Thanks for the note. I've updated the guide.

There is a bug with pyannotate: you have to manually search + replace all <locals> in the results file with (anything really, blank or whatever. It can't parse the < and this won't work anyways since it's a local).

[fennellkyle]

Thanks. There is no test/configs/osx.utsc 😞 as noted. I tried using linux.utsc instead on my Mac but it hangs shortly after ### Loading: test/tftp.uts and doesn't complete...

[gpotter2]

OSX is based on BSD 👍 there is a bsd.uts

[luizbarcelos]

Wanted to start helping on this issue, it's been a while since it was last updated though. Should I just follow the tutorial @gpotter2 ? Perhaps there is a higher priority issue for beginners on this repo, please let me know.

[gpotter2]

@luizbarcelos Hi and thanks for your interest. I try to keep this page updated so feel free to help

[Davidy22]

Hello, just grabbed the repo to take a shot at this and I opened up automaton.py to start working, then opened up ansmachine.py to see some examples and I noticed the way that type hinting is being done currently, with

    def __init__(self, **kargs):
        # type: (Any) -> None

While the type hints that I was aware of look something like:

    def __init__(self, **kargs: Any) -> None:

I'm wriitng automaton.py in the style that I'm seeing in ansmachine.py right now, but is there a reason why it's written like this instead of in PEP 484 style?

[polybassa]

Hi, thanks for your help.

It's written in comments to support older versions of python, aswell.

On October 5, 2021 3:20:17 PM GMT+02:00, David Yang @.***> wrote:

Hello, just grabbed the repo to take a shot at this and I opened up automaton.py to start working, then opened up ansmachine.py to see some examples and I noticed the way that type hinting is being done currently, with

   def __init__(self, **kargs):
       # type: (Any) -> None

While the type hints that I was aware of look something like:

   def __init__(self, **kargs: Any) -> None:

I'm wriitng automaton.py in the style that I'm seeing in ansmachine.py right now, but is there a reason why it's written like this?

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/secdev/scapy/issues/2158#issuecomment-934407858

[guedou]

With v2.5.0 being closer, can this issue be closed?

[gpotter2]

Technically there's still arch/bpf. I was secretly thinking I could finish it before 2.5.0 but it's not blocking at all. I'll close it once it's done

[KhazAkar]

Hi, Since there's plan to drop python 2.x support after 2.5.0, will functions etc get signatures not in comment-style, but directly in the future?

[gpotter2]

Yes ! that's planned

[rohanvp]

I am currently working on project hinty on file scapy/contrib/carp.py . The mypy documentation says it no longer supports python2 versions. The link https://mypy.readthedocs.io/en/latest/cheat_sheet.html from the page https://github.com/secdev/scapy/issues/2158 is going to non-existent page.

What is my way forward?

Secondly, is the following a correct process to contribute to hinty?

I have picked a file that I am working on (scapy/contrib/carp.py) . I run that file using "mypy scapy/contrib/carp.py . I get multiple errors(e.g. [name-defined] , [attr-defined] etc). I start resolving those errors one by one. Once there are no errors I continue with step 2 mentioned in https://github.com/secdev/scapy/issues/2158 . Please advise if this is the correct way.

[gpotter2]

Typing of the core is now fully achieved. I'd consider this 'completed'.

With Python 2.7 being dropped, this issue is also now obsolete. Closing