load-balance balances based on file instead of module or test

[runspired]

while it does often make sense to keep tests in a single module grouped together, a single file containing multiple modules (regardless of whether top-level or nested) will be executed by only a single browser.

[tniezurawski]

I noticed that as well and was talking about it on Emberfest. If I remember my investigation properly, it looks like files are being loaded through QUnit and requirejs. But, at the time of loading they are already splitted between containers. Then the whole file is being loaded and the modules are registered, not executed yet. Then it's up to the runner, its filtering mechanism, balancing on a fly, randomization, etc to run them.

This is especially an issue (or missed opportunity!) when running tests in multiple containers:

Assuming we have 120 test files in this project ☝ each container runs 20 test files. The files can be obviously very different:

• one has 1 module and 1 test that verifies 404 handling - very fast file
• the other has multiple modules that verify a complex screen from different angles - relatively slow file

This is where the unbalance happens. I remember @patocallaghan talking about it, and how they introduced the length limit to the test file, which for sure resulted in more test files but gave a chance for better-balanced containers and they've seen positive results (correct me if I'm wrong Pat).

Ofc, that's a workaround. To make it nicer, ember-exam (or something) should somehow change the strategy for splitting. That moment of loading files felt crucial to me but I didn't figure out how to solve it yet.

It feels that all modules have to be registered in every container so we can start balancing them on a fly or based on naive weighting (unit | integration | acceptance) or based on historical execution times. Ergo, every container should load all test files and split modules in a predictable way - so other containers will do the same. That's where "balancing them on a fly" can't happen between containers because they can't communicate with each other. But they still can within a single container where the load is spread between multiple browsers.

@runspired Did you have a chance to dig into it? What are your observations?

[runspired]

I ended up manually splitting the module in EmberData that generates 3500 tests to run across N partitions (ended up picking 20 for N). I haven't had time to think about how to redo this in exam but I think it has to center on the runner being able to dynamically adjust.

Something like: give file to run, receive list of modules in file, send back which to run. If another browser opens up tell it to open the same file and give it the next module in the list.