python
`dmypy` - poor performance when a lot of possibly unrelated errors
Bug Report
The dmypy daemon will experience poor performance if there are a lot of files with errors, even if only a single file is updated.
To Reproduce
Create a ball of mud with errors. This script requires some tweaking to really bring out the issue but its a good starting point:
https://github.com/JamesHutchison/mypy/blob/dmypy-update-perf/ball_of_mud/generate_ball_of_mud.py
The performance deficit is proportionate to the number of failing files.
Expected Behavior
dmypy is always fast for small changes
Actual Behavior
dmypy is slow.
Your Environment
- Mypy version used: 0.991
Why is this important?
When converting a large repo to mypy you will have a lot of failing files at first and you cannot use --follow-imports=silent
Notes
The cause appears to be this code:
https://github.com/python/mypy/blob/master/mypy/server/update.py#L847
From what I can tell, it looks like dmypy relies on files failing their imports to know to revisit them later. This appears to also be the reason that you cannot use --follow-imports=silent with dmypy.
For example, when --follow-imports=error:
- File A is built, and File A imports file B. File A gets errors about B not existing.
- File B is built
- File A gets built again, but this time B is cached (or loaded, or whatever) and no longer fails import
