angle-grinder icon indicating copy to clipboard operation
angle-grinder copied to clipboard

Support for applying sliding window to aggregates

Open tsutsu opened this issue 3 years ago • 0 comments

Rather than taking into account an ever-increasing number of sampled data-points, e.g. "count of requests per API key since I started aggregating", I want to instead track the count of requests per API key over the last N seconds; where API keys that stop doing anything go down to zero and then get dropped from the table.

In other words, I want to get an output somewhat equivalent to charting a Prometheus rate(foo[5m]) > 0 PromQL query, where each agrind output-line is equivalent to a PromQL output time-series.

Probably this feature would only be useful with a live log-event stream, rather than with archival logs. As such, there's probably no need to doing any complex parameterization of this hypothetical window function to take sample-times from the log-events themselves; the only parameter needed would be a literal time-interval. The window would silently evaluate now() at point of ingestion of the log-line, and attach the value to the log-event; log-events would be "in the window" as long as that attached ingestion timestamp is within the window.

tsutsu avatar Apr 21 '22 19:04 tsutsu