mypy
mypy copied to clipboard
python
Make command line tests faster
Command line tests in test-data/unit/cmdline.test are very slow since they use full typeshed stubs and run mypy in a subprocess. It looks like many of them would be easy to migrate to be normal type checker tests, or some of then could be even regular Python unit tests. Alternatively, we could perhaps use test stubs with them.
Speeding up report generation tests may also be easy.
Original slow command line test
def test_slow_command_line_test(): # Your test logic here pass
Migrated to a normal type checker test
def test_fast_type_checker_test(): # Your test logic here, without the need for subprocesses pass
Use a testing framework to parallelize tests
def run_tests_in_parallel(): import concurrent.futures tests = [test_fast_type_checker_test, test_other_fast_tests]
with concurrent.futures.ProcessPoolExecutor() as executor:
executor.map(lambda test: test(), tests)
Example of optimizing test stubs
In your stubs.py
def some_external_function(arg1, arg2) -> int: pass
Is this still the case? Runing pytest .\mypy\test\testcmdline.py runs in 51s for me; what would be considered fast? Also please let me know if this issue refers to something else than what runs when pytest .\mypy\test\testcmdline.py.
Yes, that's right / they're still slow. Due to pytest-xdist, how long they take is a function of number of cores on your machine. In CI, we have 2 cores, so pytest mypy/test/testcmdline.py -n 2 would give you a better sense of how much time these tests contribute when you open a PR
Oh hmm, actually I think Github Actions updated its runners to have 4 cores. That should be an easy win, https://github.com/python/mypy/pull/17185
Oh hmm, actually I think Github Actions updated its runners to have 4 cores. That should be an easy win.
Thanks, but what should be an easy win? Are they not being run with four workers on the GitHub instances? It's running with 8 workers on my machine even though it has 16 logical cores; do you mean tweaking that setting?
Yes, we currently run with -n 2, so with a maximum concurrency of 2. I think we set it explicitly because the value reported by os.cpu_count() on GH Actions doesn't match the actual number of cores the action has available.
Gotcha. Edit: I see @hauntsaninja submitted a PR to set n 4
But beyond that, how can we get away from running the CLI as a sub process and still test some of the stdout from the CLI? The original comment mentions converting some of the tests into regular check tests or even unit tests. Could somebody point me to examples of both?
Yeah, Jukka's main idea is to just make tests that don't actually need to be run as a subprocess into other tests. For example, I think a lot of the "config file" tests in cmdline.test could be made into normal tests (e.g. move them into check-flags.test)