Replace PyCharm code inspections by faster and easier tool?

[albertz]

One problem with the PyCharm code inspections is that they are quite slow to run, and delay the CI tests. (Related was the earlier discussion on pycharm-inspect here: #328)

Also, they might be difficult to run by others, although this has always been resolved in the end.

However, the requirements for an alternative are:

• It should actually be faster.
• It should be simple to run.
• It should cover mostly the same tests.

The last aspect is the tricky one. PyCharm goes well beyond PEP8 and checks for many standard Python conventions (many other PEPs). I don't know any other tool which even comes close.

Once we start discussing some alternatives here, I will try to extend this issue by a list of what checks we actually would want (despite PEP8). Current (incomplete) list:

• PEP8, 120 char line limit
• PEP8 naming conventions
• Docstrings must be present for all modules, public classes, public functions, public methods
• All args in a function must be documented (actually PyCharm is a bit relaxed here but we might even want to be more strict)
• Incorrect docstring, defining args which do not exist
• (All arg types and return type should be specified. Currently not enforced but we should.)
• Function default argument should not be mutable
• Incorrect types, for passed arguments to any function
• Incorrect function call, missing args or so
• Function, method, class, any global or local variable not found, not defined
• Accessing protected member
• Instance attributes defined outside __init__
• Dictionary contains duplicate keys
• Errors in string formatting
• Shadowing built-in names or names from outer scope
• Unreachable code
• Unused local variables
• Unused imports
• Potential unbound local variables
• Redundant boolean variable check
• No trailing semicolons
• ...

[JackTemaki]

Maybe Flake8 with extensions that cover the non-PEP8 cases?

[JackTemaki]

https://github.com/DmytroLitvinov/awesome-flake8-extensions

[albertz]

Note that PyCharm also covers type checks. I think the most common other tool for that is mypy.

[albertz]

Note that PyCharm also covers certain error checks, e.g. that a function must exists, or an object attrib, object method, and many more things. This is a particular important feature because this already has saved us from otherwise faulty changes multiple times. (I.e. changes which would have breaked sth but where this break was not covered by some test.)

[JackTemaki]

Note that PyCharm also covers type checks. I think the most common other tool for that is mypy.

At least for me I have type check warnings displayed for RETURNN that to do not fail the tests. I can not recommend mypy for RETURNN, as even in new projects this can cause a lot of overhead if you did not type hinted every single variable from the beginning. I needed quite the time to get used to mypy in another repository, which was definitely tiny compared to the size of RETURNN.

[JackTemaki]

There is also pylint

[albertz]

Note that PyCharm also covers type checks. I think the most common other tool for that is mypy.

At least for me I have type check warnings displayed for RETURNN that to do not fail the tests. I can not recommend mypy for RETURNN, as even in new projects this can cause a lot of overhead if you did not type hinted every single variable from the beginning. I needed quite the time to get used to mypy in another repository, which was definitely tiny compared to the size of RETURNN.

But I think it's good to have every single variable hinted. Just because it might mean lots of work does not mean we should not do it. Note that even much bigger projects like PyTorch successfully use MyPy.

Anyway, maybe MyPy is a separate discussion.

However, this issue here is about potentially replacing the PyCharm inspections. So if not MyPy, what are other alternatives which cover type checks in a similar way as PyCharm?

Actually, to be fair, PyCharm also has a couple of bugs regarding type checks, and we exclude some of the type checks in our pycharm-inspect tool.

[albertz]

Note that I tried both pylint and flake8 at some point and it was in a totally different league compared to PyCharm. Maybe the flake8 extensions you linked can somewhat catch up but I am not sure.

You will even find an initial pylintrc here in the repo. A problem also was that it had many false positives.

Also, as far as I remember, it was actually slower than PyCharm. Maybe I remember wrong. But if this is the case, it's also ruled out.

But I made this issue here also to be a bit more specific and explicitly collect cases we want to cover which are currently covered by PyCharm.

[albertz]

I now stumbled upon Ruff - an extremely fast Python linter, written in Rust. From a first glance, this looks interesting.