Pylint is painfully slow in script using gi library

[bunyk]

I run pylint from Vim using syntastic and lately it became painfully slow. For example for this ~200 lines file it takes 10 seconds to be linted:

$ python3.7 -m cProfile -s cumtime /usr/local/bin/pylint main.py
...

         14873990 function calls (12297982 primitive calls) in 9.435 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    272/1    0.002    0.000    9.436    9.436 {built-in method builtins.exec}
        1    0.000    0.000    9.436    9.436 pylint:4(<module>)
        1    0.000    0.000    9.435    9.435 __init__.py:15(run_pylint)
        1    0.000    0.000    9.075    9.075 lint.py:1371(__init__)
        1    0.000    0.000    8.978    8.978 lint.py:922(check)
        1    0.000    0.000    8.977    8.977 lint.py:1033(_do_check)
  970/934    0.001    0.000    8.685    0.009 scoped_nodes.py:599(import_module)
  983/942    0.001    0.000    8.683    0.009 manager.py:119(ast_from_module_name)
     13/7    0.000    0.000    8.500    1.214 manager.py:71(ast_from_file)
     13/7    0.002    0.000    8.499    1.214 builder.py:100(file_build)
23225/3699    0.007    0.000    8.484    0.002 {built-in method builtins.next}
8891/2559    0.005    0.000    8.468    0.003 decorators.py:126(raise_if_nothing_inferred)
8345/2556    0.011    0.000    8.466    0.003 decorators.py:83(wrapped)
    24/16    0.000    0.000    8.434    0.527 builder.py:144(_post_build)
11535/3776    0.007    0.000    8.407    0.002 util.py:144(limit_inference)
11535/3776    0.007    0.000    8.405    0.002 context.py:108(cache_generator)
7983/4336    0.010    0.000    8.317    0.002 bases.py:120(_infer_stmts)
      945    0.002    0.000    8.293    0.009 scoped_nodes.py:514(getattr)
       17    0.000    0.000    8.284    0.487 brain_gi.py:131(_import_gi_module)
 1245/797    0.002    0.000    8.279    0.010 inference.py:277(infer_attribute)
  195/168    0.001    0.000    8.236    0.049 builder.py:216(delayed_assattr)
  510/496    0.001    0.000    8.182    0.016 inference.py:822(infer_assign)
  528/516    0.001    0.000    8.178    0.016 protocols.py:318(_arguments_infer_argname)
  493/488    0.001    0.000    8.174    0.017 inference.py:248(infer_import_from)
  296/263    0.000    0.000    8.165    0.031 scoped_nodes.py:1753(_class_type)
    57/27    0.000    0.000    8.164    0.302 scoped_nodes.py:1719(_is_metaclass)
        1    0.000    0.000    8.114    8.114 lint.py:1109(get_ast)
       11    0.044    0.004    6.979    0.634 builder.py:138(string_build)
       24    0.002    0.000    5.168    0.215 builder.py:163(_data_build)
       24    0.000    0.000    3.935    0.164 rebuilder.py:147(visit_module)
       24    0.001    0.000    3.935    0.164 rebuilder.py:158(<listcomp>)
312757/2643    0.173    0.000    3.934    0.001 rebuilder.py:161(visit)
1101/1099    0.001    0.000    3.861    0.004 rebuilder.py:1030(visit_classdef)
1101/1099    0.005    0.000    3.860    0.004 rebuilder.py:388(visit_classdef)
1101/1099    0.028    0.000    3.821    0.003 rebuilder.py:404(<listcomp>)
82640/82631    0.042    0.000    3.718    0.000 rebuilder.py:606(visit_functiondef)
82649/82640    0.280    0.000    3.677    0.000 rebuilder.py:577(_visit_functiondef)
    28/27    0.000    0.000    2.066    0.077 manager.py:67(visit_transforms)
    28/27    0.000    0.000    2.066    0.077 transforms.py:83(visit)
       28    0.002    0.000    2.065    0.074 transforms.py:89(<listcomp>)
313935/3049    0.660    0.000    2.064    0.001 transforms.py:50(_visit)
1266777/8733    0.610    0.000    2.029    0.000 transforms.py:59(_visit_generic)
596488/9900    0.136    0.000    2.007    0.000 transforms.py:61(<listcomp>)

pylint --version output

pylint 2.3.0-dev1 astroid 2.2.0-dev Python 3.7.0 (default, Sep 25 2018, 19:06:08) [GCC 5.4.0 20160609]

From line 970/934 0.001 0.000 8.685 0.009 scoped_nodes.py:599(import_module) It seems to me that too much imports are done, but I'm not sure how to debug it more. Maybe I need to insert some prints in astroid to figure out what happens.

[PCManticore]

Thanks for the report! I think this is caused by two issues. First, for understanding gi we have a custom astroid transform for it: https://github.com/PyCQA/astroid/blob/master/astroid/brain/brain_gi.py. Unfortunately as you can see that is not as great as we import the gi module directly just to get access to its members. We have an issue to get rid of that import and do everything statically in https://github.com/PyCQA/astroid/issues/562

It's very possible that the slowdown you are noticing comes from that transform, can you test with that transform disabled or with gi uninstalled if you get the same performance results?

[bunyk]

Thanks for clarification. When I comment gi imports, or do pip3.7 uninstall PyGObject it decreases time from 9.4 seconds to 1 second.

About astroid transforms, I need to first figure out what that is.

[bunyk]

Not sure if this is relevant, but when I add following to my ~/.pylintrc

ignored-modules=gi,gi.repository

it does not decrease time. But "no name 'NotHere' in module gi.repository" error disappears. So it does not check existence of objects in module, but that module still somehow slows down execution. Not sure if this is another bug.

[PCManticore]

Unfortunately ignored-modules is processed after we process the astroid transform files so it does nothing to improve performance. What would improve it though is to have a static transform rather than importing that module as we currently do.

[gornostal]

Is there any workaround for this issue?

Thanks

[simonvanderveldt]

Just ran into this as well, makes it pretty much impossible to have live linting :(

@PCManticore astroid is new to me, why is gi (and pytest) an exception for astroid in that it imports the module? Is there a technical issue that prevents gi to be treated like other modules?

[PCManticore]

@simonvanderveldt That's the way the transform was written, using an import of gi. That transform seems to be the only one that imports a module like that, we should definitely rewrite the transform to be statically just like the others. This needs someone familiar with gi most likely, and I'm afraid I won't be of much help in the following period.