Optimize CI build time

[matthiasbeyer]

This might reduce build times on CI.

[matthiasbeyer]

Ah yeah, I included this patch... that should have gone to another PR, but I guess it doesn't matetr that much, does it?

Maybe there is more in terms of build time that can be done... don't know. I guess we can keep this open a bit until I figure some more optimizations out.

[matthiasbeyer]

This now depends on #1297

Either way it is just an experiment by now.

[matthiasbeyer]

Apparently, tests bind to the same address and port and thus cannot be run in parallel. Something that definitively has to be fixed.

[PradeepKiruvale]

Apparently, tests bind to the same address and port and thus cannot be run in parallel. Something that definitively has to be fixed.

So, these tests have to be run in a serial manner.

[matthiasbeyer]

Yep. Lets see what happens now, when each test has its individual broker. Would be nice if it simply worked, because this would be a huge speed improvement. Right now testing takes about 9 minutes, which is at least 7 minutes too long IMO.

[matthiasbeyer]

Okay, now I am running into test failures that seem to not have something to do with how the tests are executed (or I am just not competent enough to see it right now).

I will carve out the patches that tinker with the test running and submit them another time in a seperate PR.

I still count this experiment a success for now:

• Overall CI time is down from 21 minutes to 11 minutes
• Build time is down from 8 minutes to 40 sec (although I am not 100% sure whether this is just a perfect cache hit :laughing: )
• Test time is down from approx. 7 minutes to 5 minutes although most of that is probably the build time from building the test binaries in the job

[matthiasbeyer]

So I am quite a bit concerned that the tests are failing here.

I did not touch any actual code in this PR, but only the .github folder. That means, I did not alter tests, but only how tests are run. That should have zero impact on the results of the tests (only on the speed).

That the tests are now failing shows me that there must be some issue with how the tests are implemented.

How to proceed?

[didier-wenzek]

So I am quite a bit concerned that the tests are failing here.

Indeed a concern. I need sometimes to understand what is going on.

[reubenmiller]

Abandoned PR. Though it looked like the rust caching has already made into main.