kubeshop
SLO based Test Result Validation
Describe the solution you'd like The results of a Test Execution should be validated against a Service Level objective (SLO) may be defined in a (extended) OpenSLO YAML format. The Test results should also collect the Metrics, Traces, result sets e.g. (k6) to be validated. Especially Traces can be checked by testkube but validated against SLO?
Additional context the keptn tool has a lot of Resources for SLO Evaluation.
Hey @hwinkel for very interesting suggestion! In fact, we're planning to add keptn executor and one of our external contributors even started working on it. But you gave an intersting perspective to how we should process test execution results!
Collecting the Test Results Metrics in data driven way (Time Series DB, artefacts, traces) and comparing them to an "expected result SLO" is IMHO the most important aspect of any test automation. all results must be labeled with all availabble Meta Data like, Test Case Id, Test Run Id, Test Environment Specifics, Versions, Resources, Git SHA etc etc.. if you have this collected, you always can post process the validation and any report required. I was also thinking to integrate with Jupyter alike tools to analyse results and create report books
I think thats also a link to tracetest, which is doing that already for traces, now just add metrics, meta data and logs to the mix. With Metrics you need to consider two sets of collected Result MEtrics:
- The Metrics of the System, Device, Service under Test
- The Metrics yield by the Test Tool
Yeah, either we need to trasetesting Testkube or testkubing Tracetest a bit ;) Something serious to think about. But no doubts test metrics should be considered in a context of system/env metrics at the moment of test execution
@hwinkel Founder of Tracetest here - read your comments with interest, particularly your thoughts on Tieing the test run to metrics / logs in addition to the trace. We will be adding an executor for Tracetest to testkube - would love to have you try out Tracetest and give us some feedback! https://github.com/kubeshop/tracetest
@hwinkel I like this idea, do you have a use case in mind on how you would use this at your company ? Would love to get your feedback before planning this feature. Wanna hop in a quick call in the next days ? https://calendly.com/bruno-at-kubeshop/30min-1
