Panic in function processing (seriesaggregators.go)

[replay]

In one of our production instances we're seeing a panic occurring regularly:

[Macaron] PANIC: runtime error: index out of range [179] with length 179
/usr/local/go/src/runtime/panic.go:88 (0x434fa4)
/go/src/github.com/grafana/metrictank/expr/seriesaggregators.go:208 (0xc0b96c)
/go/src/github.com/grafana/metrictank/expr/func_aggregate.go:73 (0xbdcb8b)
/go/src/github.com/grafana/metrictank/expr/func_aggregate.go:60 (0xbdc714)
/go/src/github.com/grafana/metrictank/expr/plan.go:327 (0xc0a6a8)
/go/src/github.com/grafana/metrictank/api/graphite.go:1016 (0xc852d6)
/go/src/github.com/grafana/metrictank/api/graphite.go:318 (0xc7d424)
/usr/local/go/src/reflect/value.go:475 (0x4c11a6)
/usr/local/go/src/reflect/value.go:336 (0x4c0698)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0xb01439)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0xb00e0a)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xc6a124)
/go/src/github.com/grafana/metrictank/vendor/github.com/raintank/gziper/gzip.go:100 (0xc6a117)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 (0xb1ba92)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xb01154)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xb00ef9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xb2cda5)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:161 (0xb2cd98)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/logger.go:40 (0xb1f7b7)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xb01154)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xb00ef9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xb83d54)
/go/src/github.com/grafana/metrictank/api/middleware/logger.go:45 (0xb83d3d)
/usr/local/go/src/reflect/value.go:475 (0x4c11a6)
/usr/local/go/src/reflect/value.go:336 (0x4c0698)

I think this is an indication for a bigger issue, because actually this "index out of range" shouldn't happen, but as a quick fix we could at least add a len() check on the relevant line to prevent a panic.

[fkaleo]

I was able to reproduce it with query

diffSeries(summarize(groupByNode(stats.*.counters.statsd.metrics_received.count, 2, 'sum'), '1min', 'sum', false), summarize(timeShift(groupByNode(stats.*.counters.statsd.metrics_received.count, 2, 'sum'), '1d'), '1min', 'sum', false))

PANIC: runtime error: index out of range [60] with length 60

/usr/local/go/src/runtime/panic.go:88 (0x434fa4)
/go/src/github.com/grafana/metrictank/expr/seriesaggregators.go:208 (0xc0b96c)
/go/src/github.com/grafana/metrictank/expr/func_aggregate.go:73 (0xbdcb8b)
/go/src/github.com/grafana/metrictank/expr/func_aggregate.go:60 (0xbdc714)
/go/src/github.com/grafana/metrictank/expr/plan.go:327 (0xc0a6a8)
/go/src/github.com/grafana/metrictank/api/graphite.go:1016 (0xc852d6)
/go/src/github.com/grafana/metrictank/api/graphite.go:318 (0xc7d424)
/usr/local/go/src/reflect/value.go:475 (0x4c11a6)
/usr/local/go/src/reflect/value.go:336 (0x4c0698)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0xb01439)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0xb00e0a)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xc6a124)
/go/src/github.com/grafana/metrictank/vendor/github.com/raintank/gziper/gzip.go:100 (0xc6a117)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 (0xb1ba92)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xb01154)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xb00ef9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xb2cda5)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:161 (0xb2cd98)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/logger.go:40 (0xb1f7b7)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xb01154)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xb00ef9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xb83d54)
/go/src/github.com/grafana/metrictank/api/middleware/logger.go:45 (0xb83d3d)
/usr/local/go/src/reflect/value.go:475 (0x4c11a6)
/usr/local/go/src/reflect/value.go:336 (0x4c0698)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0xb01439)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0xb00e0a)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xb85604)
/go/src/github.com/grafana/metrictank/api/middleware/tracer.go:70 (0xb855ee)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 (0xb1ba92)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xb01154)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xb00ef9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xb84cce)
/go/src/github.com/grafana/metrictank/api/middleware/stats.go:76 (0xb84cb9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 (0xb1ba92)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xb01154)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xb00ef9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xb2cda5)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:161 (0xb2cd98)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/logger.go:40 (0xb1f7b7)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xb01154)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xb00ef9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xb1bc1c)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:187 (0xb2dff0)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:294 (0xb281cf)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/macaron.go:220 (0xb20b0d)
/usr/local/go/src/net/http/server.go:2843 (0x7971a2)
/usr/local/go/src/net/http/server.go:1925 (0x7929ac)
/usr/local/go/src/runtime/asm_amd64.s:1374 (0x46fde0)

[fkaleo]

Even simpler reproduction query:

diffSeries(SINGLE_SERIES, summarize(timeShift(SINGLE_SERIES, '-1h'), '1min'))

Also breaks with sumSeries

[fkaleo]

Panic does not always happen, only with some time ranges.

[fkaleo]

diffSeries(summarize(metrictank.stats.docker-env.default.tank.persist.values.rate32, '1min'), summarize(timeShift(metrictank.stats.docker-env.default.tank.persist.values.count32, '-1min'), '1min')) recreates the issue.

summarize(metrictank.stats.docker-env.default.tank.persist.values.count32, '1min') produces 2 data points

summarize(timeShift(metrictank.stats.docker-env.default.tank.persist.values.count32, '-1min'), '1min') produces 3 data points

[fkaleo]

diffSeries(summarize(metrictank.stats.docker-env.default.tank.persist.values.rate32, '1min'), summarize(timeShift(metrictank.stats.docker-env.default.tank.persist.values.count32, '-1min'), '1min')) recreates the issue.

reversing the arguments to diffSeries and the panic seems gone

[fkaleo]

Series showing the issue with from=1604708261266&to=1604708992552

[fkaleo]

in summarizeValues commenting out this piece fixes the issue:

	// MT specific bit: if !s.alignToFrom we want the output to be canonical
	// only thing needed is strip out the first point if its TS < from
	if len(out) != 0 && out[0].Ts < serie.QueryFrom {
		out = out[1:]
	}

[fkaleo]

very linked to previously fixed issue https://github.com/grafana/metrictank/issues/1811