flake8-isort 100 times slower than isort

[maratori]

I'd like to use flake8-isort in my project, but I can't because it is extremely slow. Here is a script to reproduce the issue.

import os
import shutil
from pathlib import Path
from time import time

def prepare_files():
    shutil.rmtree("files", ignore_errors=True)
    Path("files").mkdir(exist_ok=True)
    content = "\n".join(10000 * ["def a(): pass"])
    for i in range(100):
        Path(f"files/file{i:03d}.py").write_text(content)

def execute(cmd):
    start = time()
    os.system(cmd)
    duration = time() - start
    return duration

if __name__ == "__main__":
    prepare_files()
    isort_duration = execute("isort --check-only files")
    print(f"isort: {isort_duration:.3f} sec")
    flake8_duration = execute("flake8 --select I001,I002,I003,I004,I005 files")
    print(f"flake8: {flake8_duration:.3f} sec")
    print(f"ratio: {flake8_duration / isort_duration:.1f}")

I see following numbers on my machine (the same is on github actions).

isort: 2.511 sec
flake8: 394.109 sec
ratio: 157.0

Is it possible to do something with it?

Versions:

Python: 3.9.7
flake8: 3.9.2
flake8-isort: 4.0.0
isort: 5.9.3

[gforcada]

interesting... did you run a profiler on it to check why it is so slow? 🤔 I'm really curious on what's the slow path here...

[maratori]

Nope

[dreid]

I profiled this briefly using the first 10 files from the attached test program:

> python -m cProfile -o flake8.prof -m flake8 -- --select I001,I002,I003,I004,I005 files/file00*.py

And the following profile output would seem to indicate that isort itself, and flake8-isort are still very fast. But flake8 is still parsing every file to run all the normal flake8 checks.

The time spent under flake8_isort, run is 0.787s

       40    0.003    0.000    0.787    0.020 ve/lib/python3.8/site-packages/flake8_isort.py:190(run)

And the time spent under isort.api, sort_stream is 0.617s

       10    0.000    0.000    0.617    0.062 ve/lib/python3.8/site-packages/isort/api.py:137(sort_stream)

But a full 61s is under flake8/checker.py run_checks

       10    0.000    0.000   61.339    6.134 ve/lib/python3.8/site-packages/flake8/checker.py:593(run_checks)

And that time seems to break down mostly into, process_tokens

       10    1.327    0.133   40.254    4.025 ve/lib/python3.8/site-packages/flake8/checker.py:565(process_tokens)

and run_ast_checks.

       10    0.045    0.004   21.085    2.108 ve/lib/python3.8/site-packages/flake8/checker.py:487(run_ast_checks)

More profile output:

>>> p = pstats.Stats("flake8.prof").sort_stats('cumtime').print_stats('pyflakes|flake8|isort', 0.1)
Mon Feb 28 10:18:57 2022    flake8.prof

         105113153 function calls (103508954 primitive calls) in 64.613 seconds

   Ordered by: cumulative time
   List reduced from 1941 to 616 due to restriction <'pyflakes|flake8|isort'>
   List reduced from 616 to 62 due to restriction <0.1>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   65.145   65.145 ve/lib/python3.8/site-packages/flake8/__main__.py:1(<module>)
        1    0.000    0.000   65.118   65.118 ve/lib/python3.8/site-packages/flake8/main/cli.py:9(main)
        1    0.000    0.000   65.115   65.115 ve/lib/python3.8/site-packages/flake8/main/application.py:367(run)
        1    0.000    0.000   65.115   65.115 ve/lib/python3.8/site-packages/flake8/main/application.py:362(_run)
        1    0.000    0.000   61.470   61.470 ve/lib/python3.8/site-packages/flake8/main/application.py:256(run_checks)
        1    0.000    0.000   61.339   61.339 ve/lib/python3.8/site-packages/flake8/checker.py:297(run)
        1    0.000    0.000   61.339   61.339 ve/lib/python3.8/site-packages/flake8/checker.py:292(run_serial)
       10    0.000    0.000   61.339    6.134 ve/lib/python3.8/site-packages/flake8/checker.py:593(run_checks)
       10    1.327    0.133   40.254    4.025 ve/lib/python3.8/site-packages/flake8/checker.py:565(process_tokens)
   100000    0.149    0.000   27.190    0.000 ve/lib/python3.8/site-packages/flake8/checker.py:609(handle_newline)
   100000    4.355    0.000   27.018    0.000 ve/lib/python3.8/site-packages/flake8/checker.py:508(run_logical_checks)
  3600050    4.193    0.000   24.336    0.000 ve/lib/python3.8/site-packages/flake8/checker.py:403(run_check)
       10    0.045    0.004   21.085    2.108 ve/lib/python3.8/site-packages/flake8/checker.py:487(run_ast_checks)
       10    0.000    0.000   10.181    1.018 ve/lib/python3.8/site-packages/flake8/plugins/pyflakes.py:70(__init__)
       10    0.035    0.003   10.180    1.018 ve/lib/python3.8/site-packages/pyflakes/checker.py:896(__init__)
200010/100010    0.280    0.000    7.546    0.000 ve/lib/python3.8/site-packages/pyflakes/checker.py:1368(handleChildren)
  3600050    2.951    0.000    7.276    0.000 ve/lib/python3.8/site-packages/flake8/processor.py:242(keyword_arguments_for)
400000/300000    0.605    0.000    6.390    0.000 ve/lib/python3.8/site-packages/pyflakes/checker.py:1402(handleNode)
   700010    0.538    0.000    4.777    0.000 ve/lib/python3.8/site-packages/flake8/processor.py:267(generate_tokens)
   100000    0.198    0.000    4.415    0.000 ve/lib/python3.8/site-packages/pyflakes/checker.py:2103(FUNCTIONDEF)
   700000    0.585    0.000    4.187    0.000 ve/lib/python3.8/site-packages/flake8/checker.py:622(check_physical_eol)
        1    0.000    0.000    3.476    3.476 ve/lib/python3.8/site-packages/flake8/main/application.py:353(report)
        1    0.000    0.000    3.476    3.476 ve/lib/python3.8/site-packages/flake8/main/application.py:301(report_errors)
        1    0.001    0.001    3.476    3.476 ve/lib/python3.8/site-packages/flake8/checker.py:233(report)
       12    0.130    0.011    3.387    0.282 ve/lib/python3.8/site-packages/flake8/checker.py:151(_handle_results)
       20    0.093    0.005    3.262    0.163 ve/lib/python3.8/site-packages/pyflakes/checker.py:948(runDeferred)
   200032    0.131    0.000    3.258    0.000 ve/lib/python3.8/site-packages/flake8/style_guide.py:397(handle_error)
       20    0.092    0.005    3.245    0.162 ve/lib/python3.8/site-packages/flake8/processor.py:125(file_tokens)
   200032    0.543    0.000    3.127    0.000 ve/lib/python3.8/site-packages/flake8/style_guide.py:530(handle_error)
   100000    0.166    0.000    3.064    0.000 ve/lib/python3.8/site-packages/pyflakes/checker.py:2169(runFunction)
   100000    0.487    0.000    2.898    0.000 ve/lib/python3.8/site-packages/flake8/checker.py:534(run_physical_checks)
   700000    1.392    0.000    2.608    0.000 ve/lib/python3.8/site-packages/flake8/processor.py:417(log_token)
   100000    0.427    0.000    2.530    0.000 ve/lib/python3.8/site-packages/pyflakes/checker.py:2117(LAMBDA)
   100000    1.210    0.000    1.941    0.000 ve/lib/python3.8/site-packages/pyflakes/checker.py:784(in_annotation_func)
       10    0.000    0.000    1.910    0.191 ve/lib/python3.8/site-packages/flake8/processor.py:221(build_ast)
       10    0.000    0.000    1.847    0.185 ve/lib/python3.8/site-packages/flake8_plugin_utils/plugin.py:68(run)
       10    0.099    0.010    1.769    0.177 ve/lib/python3.8/site-packages/pyflakes/checker.py:829(_collect_type_comments)
   101570    0.600    0.000    1.650    0.000 ve/lib/python3.8/site-packages/pyflakes/checker.py:1119(addBinding)
   100000    0.075    0.000    1.542    0.000 ve/lib/python3.8/site-packages/pyflakes/checker.py:818(_typeable)
   200032    0.187    0.000    1.484    0.000 ve/lib/python3.8/site-packages/flake8/formatting/base.py:85(handle)
   200032    0.296    0.000    1.443    0.000 ve/lib/python3.8/site-packages/flake8/checker.py:382(report)
   100000    0.163    0.000    1.174    0.000 ve/lib/python3.8/site-packages/flake8/processor.py:225(build_logical_line)
   200030    0.542    0.000    1.007    0.000 ve/lib/python3.8/site-packages/flake8/processor.py:280(noqa_line_for)
   500010    0.616    0.000    0.999    0.000 ve/lib/python3.8/site-packages/pyflakes/checker.py:259(iter_child_nodes)
   100000    0.753    0.000    0.952    0.000 ve/lib/python3.8/site-packages/flake8/processor.py:186(build_logical_line_tokens)
   200032    0.082    0.000    0.928    0.000 ve/lib/python3.8/site-packages/flake8/formatting/base.py:189(write)
   200032    0.205    0.000    0.846    0.000 ve/lib/python3.8/site-packages/flake8/formatting/base.py:182(_write)
       40    0.003    0.000    0.787    0.020 ve/lib/python3.8/site-packages/flake8_isort.py:190(run)
  3600000    0.758    0.000    0.779    0.000 ve/lib/python3.8/site-packages/flake8/processor.py:166(update_checker_state_for)
   300000    0.269    0.000    0.670    0.000 ve/lib/python3.8/site-packages/pyflakes/checker.py:961(futuresAllowed)
   100000    0.050    0.000    0.640    0.000 ve/lib/python3.8/site-packages/pyflakes/checker.py:2197(ARGUMENTS)
       10    0.000    0.000    0.617    0.062 ve/lib/python3.8/site-packages/isort/api.py:137(sort_stream)
       10    0.359    0.036    0.614    0.061 ve/lib/python3.8/site-packages/isort/core.py:29(process)
   700000    0.452    0.000    0.556    0.000 ve/lib/python3.8/site-packages/flake8/processor.py:393(is_eol_token)
   200032    0.201    0.000    0.514    0.000 ve/lib/python3.8/site-packages/flake8/statistics.py:30(record)
   200010    0.443    0.000    0.467    0.000 ve/lib/python3.8/site-packages/pyflakes/checker.py:1347(_handle_type_comments)
   200032    0.329    0.000    0.329    0.000 ve/lib/python3.8/site-packages/flake8/formatting/default.py:30(format)
    99990    0.075    0.000    0.329    0.000 ve/lib/python3.8/site-packages/pyflakes/checker.py:1108(differentForks)
   200032    0.148    0.000    0.276    0.000 ve/lib/python3.8/site-packages/flake8/style_guide.py:72(is_inline_ignored)
   100000    0.104    0.000    0.270    0.000 ve/lib/python3.8/site-packages/pyflakes/checker.py:1494(handleAnnotation)
    99990    0.097    0.000    0.263    0.000 ve/lib/python3.8/site-packages/pyflakes/checker.py:1058(report)

[akx]

@dreid So IOW, it sounds like flake8 isn't smart enough to realize that no checks are enabled that would actually require it (or pyflakes) parsing the file...

[gforcada]

Meanwhile, if you want something fast, you can use ruff 😄