bioyino icon indicating copy to clipboard operation
bioyino copied to clipboard

statsd compatible rate aggregation for counter

Open Karsonito opened this issue 1 year ago • 0 comments

Hello! I seem to have found an inconsistency in the implementation of rate aggregation for counters. Bioyino calculates rate as "counts per sec", while statsd implies "value per sec" Not to be confused with timer's count_ps in statsd (also called rate in bioyino) - it's different aggregation

Can we make it compatible with statsd? https://github.com/statsd/statsd/blob/7c07eec4e7cebbd376d8313b230cea96c6571423/lib/process_metrics.js#L19

How to reproduce:

  • Prepare instances with such configs:

statsd

{
, flushInterval: 10000
, deleteCounters: true
, graphite: {
    legacyNamespace: false
  , globalPrefix:    ""
  , prefixCounter:   ""
  }
}

bioyino

[carbon]
interval = 10000

[aggregation]
round-timestamp = "down"
update-count-threshold = 0
aggregates.counter = [ "value", "rate", "updates" ]

[naming.default]
destination = "name"
prefix = ""

[naming.counter]
postfixes = { "value" = "count" }
  • And compare results:

Samples:

for v in {1..100}; do echo "test.counter:$v|c" | ncat -u 127.0.0.1 51125; echo "test.counter:$v|c" | ncat -u 127.0.0.1 52125; done;

Results:

statsd

ncat -klp 51003 | grep -P "^test"
test.counter.rate 505 1704117415
test.counter.count 5050 1704117415

bioyino

cat -klp 52003 | grep -P "^test"
test.counter.count 5050.0 1704117410
test.counter.rate 10.0 1704117410
test.counter.updates 100.0 1704117410

As we see test.counter.rate uses different logic to calculate.

I see that rate was asked ealier in issue https://github.com/avito-tech/bioyino/issues/37 as

rate as sum / (carbon.interval / 1000)

but implemented only

sample_rate as agg.len() / (carbon.interval / 1000)

Later we discussed rate in implementation of sample rate https://github.com/avito-tech/bioyino/issues/53#issuecomment-805937185

Real events count inside app: 300 Sample rate: 0.1 Sent\received count: 30 Interval: 30 Aggregates: sum = 30/0.1 = 300, rate = sum / 30 = 10

My fault for not checking rate for counter

Karsonito avatar Jan 01 '24 14:01 Karsonito