Improve performance

[arnaud-ma]

The most expensive thing (by far) in all the code is parsing the comments of the class variables, and in particular getting the source code of the classes with inspect.getsource and tokenize.generate_tokens.

I noticed 2 bottlenecks about this:

• The biggest: issubclass(Tap, Tap) is True! This caused both functions to be called for Tap itself, which is quite large and so they had trouble
• The two functions where called many times for one class. I suspect there is already some caching involved on the inspect side anyway, but it will always be better not to count on that. The idea is to replace the obj parameter in each parsing function into what they really need (the tokens or the source code).

Benchmark

Used ipython for the timeit

With a simple class like this:

from tap import Tap

class Foo(Tap):
    name: str  # Your name

Before:

In [2]: %timeit Foo()
15.1 ms ± 306 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)

After:

In [2]: %timeit Foo()
239 μs ± 4.25 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

Of course, the bigger the class, the slower it will be:

class Foo(Tap):
    name: str  # Your name
    language: str = "Python"  # Programming language
    package: str = "Tap"  # Package name
    stars: int  # Number of stars
    max_stars: int = 5  # Maximum stars

In [2]: %timeit Foo()
404 μs ± 4.25 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

It will always be slower than argparse.ArgumentParser:

from argparse import ArgumentParser

def arg_parser():
    parser = ArgumentParser()
    parser.add_argument("--name", type=str, required=True, help="Your name")
    parser.add_argument(
        "--language",
        type=str,
        default="Python",
        help="Programming language",
    )
    parser.add_argument("--package", type=str, default="Tap", help="Package name")
    parser.add_argument("--stars", type=int, required=True, help="Number of stars")
    parser.add_argument("--max_stars", type=int, default=5, help="Maximum stars")

In [2]: %timeit arg_parser()
113 μs ± 664 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

But it's quite close, i don't think it is an issue now.. as long as the class is small. I don't think that's enough to solve #42.

[codecov-commenter]

:warning: Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 84.21053% with 3 lines in your changes missing coverage. Please review.

Project coverage is 93.10%. Comparing base (c0d4b75) to head (09cb610). Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
src/tap/utils.py	82.35%	3 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #149      +/-   ##
==========================================
- Coverage   93.49%   93.10%   -0.39%     
==========================================
  Files           4        4              
  Lines         722      725       +3     
==========================================
  Hits          675      675              
- Misses         47       50       +3     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[arnaud-ma]

Ah, the coverage diff is because the get_subsequent_assign_lines and source_line_to_tokens functions were not explicitly tested before. Now that they are no longer used in the code, it only shows up here

[martinjm97]

Feel free to remove code that is no longer used in the PR.

[martinjm97]

This is wonderful! Thank you for improving not only the performance, but also the code quality along the way. Much appreciated!

--JK