Improve testsuite performance (speed, parallelization) [33]

[brejoc]

Salt TestSuite Documentation: https://docs.saltstack.com/en/latest/topics/development/tests/index.html#salt-s-test-suite

Checkout: https://github.com/saltstack/salt/pull/53566

Speed improvements

• [x] identify slow tests (use profiler)
• [ ] Investigate the reason of the slowest tests: https://github.com/saltstack/community/issues/95
• [ ] create PR to improve performance

Parallelization

• [ ] research how to group the tests in relevant units
• [ ] create PoC of running the groups in parallel
• [ ] present results

[waynew]

Here's the slowest tests - about 1,000 - from Salt (pulled from Jenkins).

SaltStack tests that never ran _1s - Tests to skip (_1s).csv.txt

For each of the platforms that we test (see Jenkins), these are the tests that never ran <1s. Every single test took >=1s to run. I included the max and min test times, since some of the tests ran faster on one platform than the other. The times are in seconds.

This represents about 10% of the tests that are executed on any platform.

For some extra flavor, here's another breakdown by test times:

The legend on the right is the threshold in seconds. About 75% of tests take <0.01s, and then about another ~7% of tests <0.1s, <1s respectively, to get ~90% of tests <1s.