xk6-output-prometheus-remote
xk6-output-prometheus-remote copied to clipboard
Investigate potential for concurrency processing
There are two main ways to add concurrency to the extension:
- concurrency at the level of processing metrics pre-request
Details: Output receives batches of metrics that must be iterated over and converted into remote write TimeSeries. This may seems as a natural point to add concurrency like this:
samplesContainers := o.GetBufferedSamples()
step := math.Floor(len(samplesContainers) / concurrencyLimit)
for i := 0; i < concurrencyLimit; i++ {
wg.Add(1)
// get chunk of samplesContainers from i * step to (i+1) * step
go func(...) {
...
gatherpoint[i] = convertToTimeSeries(chunk)
...
}(...)
}
wg.Wait()
for i := 0; i < concurrencyLimit; i++ {
allTS = append(allTS, gatherpoint[i]...)
}
// encode and send remote write request
But this processing must be done within 1 second of flush period. Basic experiments so far showed next to none improvement in trying to spawn goroutines within that time limit. This result will likely be impacted by changes from Metric Refactoring in k6 and might need more investigation.
- concurrency at remote write requests
Details: this is blocked by inability to compile TimeSeries (group samples). Attempt to send disjointed samples concurrently would only result in out of order errors.