Notice: Number of Test Permutations are Being Capped

[NateBrady23]

We are currently reviewing official runs from Azure and Citrine as Round 20 candidates. TFB-status should be back online tomorrow for others to also review those results before we complete the process.

I've mentioned this in a couple pull requests over the last few months, but we will be capping the number of test permutations a framework can have. That number is tentatively 15. This isn't the number of individual test types, but the number of permutations: think "nodejs-mongo", "nodejs-postgres", etc. as a single permutation.

The reason for this is the amount of time it takes to build and test each framework is considerable. Some frameworks with 25 or more permutations can hold onto our benchmarking resources for long amounts of time, even with docker caching, etc. As new frameworks are added regularly, we want to make sure that benchmark runs can finish in a reasonable amount of time and don't spend several hours on one framework.

With that said, maintainers that wish to test all kinds of different permutations are welcome to open as many pull requests as they like switching permutations out between runs as long as a single run doesn't exceed 15 permutations. The new config files coming shortly will also support commenting out tests so you can do this more easily.

Thanks for understanding and may your requests per second always improve!