ThrowTheSwitch
Parallel Test Build And Run
I was wondering whether it is possible to parallelise the test build and run process?
I've had a look through the code and there are hints at parallel build for release, but not for test. With my extremely limited knowledge of rake & ruby I've tried basic attempts at parallelising stuff from the top level, but I think there are more intricacies than I understand to this - I just end up with funny errors which I presume are things interacting strangely.
On our project I've enabled the rake multitasking by default which shaved ~90s off our original test time (now we're at ~7 minutes for close to 400 tests - and growing), but I think we could make further gains by parallelising the test run which seem to be single threaded at the moment.
Are there any easy wins we can make here or is it quite a fundamental change to the infrastructure?
I'd be interested in seeing this, however I'm not sure how much can be done asynchronously as is. A lot of ceedling is shell commands and checking the return of them and the final testing collection is collecting the output of the runners to determine what passed and what didn't.
If we do all the building of mocks first, followed by building of needed modules, we could potentially put these in parallel and that should be a significant performance improvement without doing everything in parallel.
I'd be down for attempting that, but the amount of work needed would mean a lot of changes that we need a good way to handle.
Hey, how about something like this? At least with my setup I did not encounter any problems.
Small benchmark from one of my projects:
| Test | Time in seconds |
|---|---|
| Unmodified | 161,494250 |
| 1 Thread | 161,024545 |
| 4 Threads | 55,5480937 |
| 8 Threads | 41,3095573 |
As an update to this, I copied over the updates the pJunger did to v0.28.3 and there appears to be some issues with how mocks are created. Basically a bunch of tests fail to compile and from what I can tell this is because all the mocks are created on the fly. So if two threads try to create/modify/use the same mock at the same time a bunch of errors occur.
My guess is that you'd really need either pre-process all the files and create the resources before going parallel, or to create separate build folders for each thread, then copy the results.
Could also be I have no idea what I'm doing :)
Any update on this? I've testing on:
Ceedling:: 0.28.2
CException:: 1.3.1.18
CMock:: 2.4.4.215
Unity:: 2.4.1.120
Before any changes:
OVERALL TEST SUMMARY
TESTED: 200 PASSED: 199 FAILED: 1 IGNORED: 0
Running in WSL I had it running in:
clean build:
real 4m17.984s
user 0m20.297s
sys 3m45.250s
rebuild 1:
real 1m50.167s
user 0m6.063s
sys 1m40.156s
When using this @pJunger code I have the same problem of mocks not being generated and used correctly. I tested on both windows and also in Ubuntu using WSL.
Running in windows causes lots of weird compiling errors. Running in WSL causes less compiling errors but still lots of tests didn't work.
Setting the config:
:project:
:compile_threads: 1
:test_threads: 1
Seems to make the code actually take longer to compile and test than without this code for some reason, and on my system fails some of the jobs. In the summary it shows just over 2x more tests running than I actually have. This seems weird.
I get:
And a range of the tests never even ran.
OVERALL TEST SUMMARY
TESTED: 446 PASSED: 440 FAILED: 6 IGNORED: 0
If I do a test on a single module in ceedling then it detects the correct number of tests. It is ceedling test:all that must be running tests more times than required for some reason?
@DPastl did you try:
:project:
:compile_threads: 1
:test_threads: 4
This doesn't have the problem you were having of the mocks being rewritten mid compile since the compiling is all then done in a single thread. This does allows the tests to run in parallel. Still doesn't work for me since there is something else going wrong, but would be interested if it works for you.
Whoa, sorry for the delay in responding, I'll give that a shot and let you know.
I think part of my problem is that for some reason my Windows Box is UNBELIEVABLY slow running the unit tests. Switch to Linux and it's a 10x speedup at least. That actually solved my problem more than anything, now I just do Unit testing in Linux.
Linux: 12.6s Windows: 148.8s
About the problems in my branch: I don't think I added any synchronization to it, as it was just an experiment to see what could be done performance wise. I agree that the creation of mocks etc. should be serialized (or at least there should be a synchronized list of mocks that then could be generated in parallel).
Regarding bad performance on windows: this sounds suspiciously like a Windows defender issue.
Thank @pJunger , I gave disabling defender a shot but that didn't work. Could be the build tools, since it's compiling to 32bit on a 64bit machine. Or maybe it's a feature in Windows ;).
Hello all, is there any update about that? I am vary interested in this feature ( > 500 unit tests).
Also interested in this. I realize now that I may have wasted some time in getting my simulator test fixture to be able to be parallelized 🤦♂️
