Allow limiting the number of concurrent TestRuns (maybe even per "target" or "SUT")

[frittentheke]

Feature Description

Currently the operator works on all TestRun resources and moves them through the state machine

• initialization
• initialized
• created
• started
• stopped
• finished
• error

It might make sense to allow restricting the number of concurrent TestRuns to avoid interference of individual runs. This might be due to noisy neighbors issue on the K8 cluster / infrastructure running the K6 pods but more importantly in cases where the same system under test (SUT) is targeted by multiple TestRuns.

Suggested Solution (optional)

1. Introduce a setting to limit the number of concurrently executed TestRuns.

• Extend the TestRunStatus with pending or queued to indicate back to the user that the TestRun is waiting for others to finish.
• Ensure first-come-first-served by sorting by TestRun creation date!
• Maybe it makes sense to already run the initializer pod to report back issues with the tests without making the user wait for the actual run.

2. The use-case I have in mind is about limiting the number of concurrent tests hitting the same system under test / target. So to extend on the limitation of concurrent TestRuns it would also be beneficial to allow to set a limit per "SUT". Either by extending the TestRun CR with a field to indicate which target will be hit or by an annotation. Yes, the real world is always more complex, so there might be even more rules that some environments would like followed.

But by allowing the k6-operator to work through a list (~ queue) of TestRuns one by one per SUT seems to cover the 90%-use-case of avoiding interference of tests (towards the same target)

Already existing or connected issues / PRs (optional)

No response

[yorugac]

Hi @frittentheke, thanks for the issue! This is an interesting idea :slightly_smiling_face: Though I suspect we wouldn't want that kind of configuration as part of TestRun CRD. Maybe as part of PrivateLoadZone or even some new CRD.

Could you please elaborate on what you mean by "interference of individual runs"? And also, why it can't be achieved through external (to k6-operator) tooling?

[frittentheke]

This is an interesting idea 🙂 Though I suspect we wouldn't want that kind of configuration as part of TestRun CRD. Maybe as part of PrivateLoadZone or even some new CRD.

I actually thought of this an an operator config option to limit the number of concurrent TestRuns that the operator creates Jobs and Pods for. Right how the operator will simply produce Jobs and Pods for all TestRun CRs that exist and I'd like a simple option like --concurrent-testruns

Could you please elaborate on what you mean by "interference of individual runs"?

This is two-fold:

1. Interference of TestRuns due to the used infrastructure (e.g. the same Kubernetes cluster) to run the k6 Pods on
2. Interference of TestRuns due to them targeting the same SUT. So running them both at the same time would mess up the test results.

And also, why it can't be achieved through external (to k6-operator) tooling?

[dgzlopes]

Hello! I found out that Kubernetes, since v1.15, supports ResourceQuota

With this you can limit the amount of resources in a specific namespace... and it works with CRDs. Here is an example:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: basic-quota
  namespace: plz-ns
spec:
  hard:
    pods: "2"
    count/privateloadzones.k6.io: "1"
    count/testruns.k6.io: "1"

With this, you could limit he number of TestRuns in a specific namespace. It isn't very flexible, but at least it is something 😄