TechEmpower
Cutelyst exceeds number of test mutations
According to https://github.com/TechEmpower/FrameworkBenchmarks/issues/8420 test mutations should be limited.
The number of test mutations will be limited to 10.
Cutelyst has 14 test mutations: https://github.com/TechEmpower/FrameworkBenchmarks/blob/f99bb89276455eb3f90eb3cac8253bfa70b73af3/frameworks/C%2B%2B/cutelyst/benchmark_config.json
As Cutelyst also has build times of 45 minutes it is using up a disproportionate amount of time. Maybe we should limit the amount of time per framework?
Let's ping @dantti. Also, it seems that the DB and the non-DB tests are covered by different configurations, which could probably be combined.
As Cutelyst also has build times of 45 minutes it is using up a disproportionate amount of time.
The issue here is lacking of docker caching, building the real code shouldn't take more than one minute.
Also, it seems that the DB and the non-DB tests are covered by different configurations, which could probably be combined.
The config_socket.ini is not used anymore, I should remove it, uwsgi support was removed as it was slower and an additional dep.
My own profiling with PR #10125 showed that by far the slowest step was apt install. I tried an optimization in #10173 (use buildpack-deps instead of ubuntu as the base image), but there hasn't been a complete run with it yet, so that I could look at the logs and check if it has had any effect; YMMV.
BTW the issue was completely unreproducible locally and in CI; it seems to affect specifically the continuous benchmarking environment.
P.S. The optimization shaved off around 10 minutes off the build time, which is a pretty good result.