Rethink test distribution across GitHub Actions instances
Our current test distribution strategy is to group the tests by name: https://github.com/mozilla/OpenWPM/blob/01f399328582041436eeb40844b0a4687a1d0561/.travis.yml#L3-L14
However this leads to an uneven distribution of tests across the runners resulting in our CI taking longer to show the final results as it is waiting for one runner to finish up.
My suggestion:
Run all the tests locally with --durations=0 to get the duration of each test. Then start distributing them in an iterative fashion across the different runners.
E.g. if we had
- test_a
- test_b
- test_c
- test_d ...
We'd end up with the following distribution:
| Runner 1 | Runner 2 | Runner 3 | Runner 4 |
|---|---|---|---|
| test_ a | test_b | test_c | test_d |
| test_ e | test_f | test_g | test_h |
| .... |
We should bundle up the functionality to parse the durations report into a list and then generating the new env variables as a script. This way we can run it every release and whenever we add a lot of tests.