OpenWPM icon indicating copy to clipboard operation
OpenWPM copied to clipboard

Rethink test distribution across GitHub Actions instances

Open vringar opened this issue 5 years ago • 0 comments

Our current test distribution strategy is to group the tests by name: https://github.com/mozilla/OpenWPM/blob/01f399328582041436eeb40844b0a4687a1d0561/.travis.yml#L3-L14

However this leads to an uneven distribution of tests across the runners resulting in our CI taking longer to show the final results as it is waiting for one runner to finish up.

My suggestion: Run all the tests locally with --durations=0 to get the duration of each test. Then start distributing them in an iterative fashion across the different runners. E.g. if we had

  1. test_a
  2. test_b
  3. test_c
  4. test_d ...

We'd end up with the following distribution:

Runner 1 Runner 2 Runner 3 Runner 4
test_ a test_b test_c test_d
test_ e test_f test_g test_h
....

We should bundle up the functionality to parse the durations report into a list and then generating the new env variables as a script. This way we can run it every release and whenever we add a lot of tests.

vringar avatar Nov 19 '20 16:11 vringar