astroid icon indicating copy to clipboard operation
astroid copied to clipboard

Cache find_module calls

Open correctmost opened this issue 1 year ago • 3 comments

Type of Changes

Type
:hammer: Refactoring

Description

This PR speeds up the linting of codebases with lots of imports like yt-dlp.

Stats

Before

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    63840    0.702    0.000    3.424    0.000 astroid/interpreter/_import/spec.py:129(find_module)
    38635    0.057    0.000    1.358    0.000 astroid/interpreter/_import/spec.py:287(find_module)
    38635    0.056    0.000    0.711    0.000 astroid/interpreter/_import/spec.py:224(find_module)
    38635    0.052    0.000    1.774    0.000 astroid/interpreter/_import/spec.py:263(find_module)
Command Mean [s] Min [s] Max [s] Relative
pylint --recursive=y . 32.063 ± 0.137 31.820 32.287 1.00

After

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    18166    0.226    0.000    1.194    0.000 astroid/interpreter/_import/spec.py:129(find_module)
     6689    0.013    0.000    0.629    0.000 astroid/interpreter/_import/spec.py:225(find_module)
     6689    0.011    0.000    0.278    0.000 astroid/interpreter/_import/spec.py:290(find_module)
     6689    0.009    0.000    0.311    0.000 astroid/interpreter/_import/spec.py:265(find_module)
Command Mean [s] Min [s] Max [s] Relative
pylint --recursive=y . 30.058 ± 0.142 29.917 30.283 1.00

correctmost avatar Sep 05 '24 22:09 correctmost

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 93.24%. Comparing base (71f5c0c) to head (a01c9da). Report is 118 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2543      +/-   ##
==========================================
+ Coverage   92.98%   93.24%   +0.26%     
==========================================
  Files          93       93              
  Lines       11047    11060      +13     
==========================================
+ Hits        10272    10313      +41     
+ Misses        775      747      -28     
Flag Coverage Δ
linux 93.13% <100.00%> (+0.27%) :arrow_up:
pypy 93.24% <100.00%> (+0.26%) :arrow_up:
windows 93.24% <100.00%> (+0.27%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
astroid/interpreter/_import/spec.py 97.58% <100.00%> (+0.08%) :arrow_up:
astroid/manager.py 89.95% <100.00%> (+0.12%) :arrow_up:

... and 59 files with indirect coverage changes

codecov[bot] avatar Sep 05 '24 22:09 codecov[bot]

Any further changes required for this PR? Thanks!

correctmost avatar Oct 21 '24 18:10 correctmost

Thank you for the reminder. I think it's fine, I prefer to have a second maintainer's opinion on some MR if possible though :) And I forgot to add a milestone so we didn't merge it for the 3.3.0 release.

Pierre-Sassoulas avatar Oct 21 '24 18:10 Pierre-Sassoulas