influxdb icon indicating copy to clipboard operation
influxdb copied to clipboard

aggregateWindow with first/last is very slow

Open gisakuya opened this issue 3 years ago • 4 comments

syntax: group |> aggregateWindow, when use with mean|avg|sum,the reponse is very fast,but slow with frist|last。 I note that when use with mean|avg|sum, it will use a pushdown pattern,but frist|last doesn't

use with sum|mean|avg image

use with frist|last image

os: windows server version: influxdb 2.5

gisakuya avatar Dec 14 '22 09:12 gisakuya

Can confirm. This issue makes it really difficult to author correct dashboards for which downsampling by a mean or such is not appropriate. Even things like min or max fail in the same way, even though they are conceptually really close to sum. It seems like this aggregateWindow function just happens to rely on peephole optimizations to work.

nagisa avatar Jun 20 '23 18:06 nagisa

I stumbled upon this, as well. For a reasonably large data set (in my case, several months of ~10 Hz data), aggregating by mean is nice and snappy, but max, last, and median all result in a timed out query. Please forgive the lack of understanding of what is going on under the hood, but I'm naively surprised that mean is so much faster than last, since mean has to operate on every record in the interval, whereas last or first does not. I guess that's the difference between a pushdown and doing the operation in memory.

I know that Flux is end of life, but can anyone speak to whether this is a Flux behavior specifically, or is it an InfluxDB issue? Does the same behavior exist with InfluxQL? To second @nagisa, it's really unfortunate that this happens, because there are lots of cases where downsampling by mean is inappropriate, and last or max is necessary to be able to interpret the data correctly.

drcraig avatar Mar 01 '24 07:03 drcraig

FWIW, I'm still on InfluxDB v2.5.1, so I can't speak to the behavior on 2.7. The docs as of v2.7 would seem to indicate that max is included in the pushdown, based on the example shown there. But I didn't see anything in the release notes between 2.5 and 2.7 referring to pushdowns or aggregation. The last mention of a pushdown change is when mean was added in v2.0.0-rc3.

drcraig avatar Mar 01 '24 20:03 drcraig

Just in case anyone is wondering, the docs history indicates that max, min, last, first, etc. were documented as being pushdowns in v2.5.

drcraig avatar Jul 23 '24 23:07 drcraig