brimdata
"panic: index out of range" when yielding variable inside lateral subquery containing sort+head
tl;dr
$ echo '{a:[1,2],c:1} {a:[3,4],c:1}' | super -z -c "over a with c => (sort | head 1 | yield {c})" -
panic: runtime error: index out of range [0] with length 0
goroutine 1 [running]:
runtime/debug.Stack()
/usr/local/opt/go/libexec/src/runtime/debug/stack.go:26 +0x5e
github.com/brimdata/super/runtime/sam/op.(*Catcher).Pull.func1()
/Users/phil/work/super/runtime/sam/op/catcher.go:25 +0x3d
panic({0x1076fb20?, 0xc000046bd0?})
/usr/local/opt/go/libexec/src/runtime/panic.go:792 +0x132
github.com/brimdata/super/runtime/sam/expr.Var.Eval(...)
/Users/phil/work/super/runtime/sam/expr/var.go:14
github.com/brimdata/super/runtime/sam/expr.(*recordExpr).Eval(0xc00037bdb0, {0x589a7d38, 0xc00016bec0}, {{0x1086a360?, 0x119ed3e0?}, 0xc00070ecc8?, 0x11b9b5c0?})
/Users/phil/work/super/runtime/sam/expr/values.go:52 +0x14c
github.com/brimdata/super/runtime/sam/op/yield.(*Op).Pull(0xc0000bdd00, 0x0)
/Users/phil/work/super/runtime/sam/op/yield/yield.go:34 +0x42b
github.com/brimdata/super/runtime/sam/op/traverse.(*Exit).pullPlatoon(0xc00016b950)
/Users/phil/work/super/runtime/sam/op/traverse/scope.go:222 +0x2f
github.com/brimdata/super/runtime/sam/op/traverse.(*Exit).Pull(0xc00016b950, 0x60?)
/Users/phil/work/super/runtime/sam/op/traverse/scope.go:208 +0x2b
github.com/brimdata/super/runtime/sam/op.(*Single).Pull(0xc00016b9b0, 0x5?)
/Users/phil/work/super/runtime/sam/op/mux.go:120 +0x33
github.com/brimdata/super/runtime/sam/op.(*Catcher).Pull(0xabd8fbab79191e99?, 0x39?)
/Users/phil/work/super/runtime/sam/op/catcher.go:28 +0x5c
github.com/brimdata/super/runtime/exec.(*Query).Pull(0xe8ba0df?, 0x0?)
/Users/phil/work/super/runtime/exec/query.go:49 +0x3c
github.com/brimdata/super/zbuf.CopyMux(0xc0004f9ca8, {0x10861580, 0xc00016b9e0})
/Users/phil/work/super/zbuf/mux.go:40 +0x38
github.com/brimdata/super/cmd/super/root.(*Command).Run(0xc0003ae000, {0xc00003c090, 0x1, 0x1})
/Users/phil/work/super/cmd/super/root/command.go:167 +0x925
github.com/brimdata/super/pkg/charm.path.run({0xc000114008, 0x1, 0x1}, {0xc00003c090, 0x1, 0x0?})
/Users/phil/work/super/pkg/charm/path.go:11 +0x7b
github.com/brimdata/super/pkg/charm.(*Spec).Exec(0x119b2200, {0xc00003c060, 0x4, 0x4})
/Users/phil/work/super/pkg/charm/charm.go:71 +0x105
main.main()
/Users/phil/work/super/cmd/super/main.go:40 +0x5b
Details
Repro is with super commit 9f87359. I bumped into this while trying to assemble a workaround to #5584.
In this simplified query, my goal is to yield the first value in the sorted contents of each array a along with another value from the outer record. If manipulate only the array portion, it runs ok.
$ super -version
Version: v1.18.0-450-g9f873597e
$ echo '{a:[1,2],c:1} {a:[3,4],c:1}' | super -z -c "over a with c => (sort | head 1 | yield {a:this})" -
{a:1}
{a:3}
I can also touch the c value via variable reference as long as I don't try to manipulate the array value at the same time.
$ echo '{a:[1,2],c:1} {a:[3,4],c:1}' | super -z -c "over a with c => (yield {c})" -
{c:1}
{c:1}
{c:1}
{c:1}
It also runs ok if I drop the sort.
$ echo '{a:[1,2],c:1} {a:[3,4],c:1}' | super -z -c "over a with c => (head 1 | yield {a:this,c})" -
{a:1,c:1}
{a:3,c:1}
But once I do anything that combines the sort+head+referencing the variable c, I see the panic like the one I showed at the top of the issue.
In reviewing this issue, @mccanne spotted that this is likely a problem with the the variable reference to c not making it through sort. He acknowledged this is a bug that could be fixed, but he also noted that we have hopes of doing some more fundamental adjustments to over before we finalize the first GA release of SuperDB, and the query shown here might look different after that work is done. Therefore I'll hold this one open pending the resolution of that effort and perhaps it can be closed then.
Verified in super commit f016b90.
The replacement of the prior over functionality with new unnest functionality as added in #6021 removed the dependency on variables and so this bug no longer occurs. Rewriting the original repro query using current syntax:
$ super -version
Version: f016b909b
$ echo '{a:[1,2],c:1} {a:[3,4],c:1}' | super -s -c "unnest {c,a} into (sort a | head 1 | values {a,c})" -
{a:1,c:1}
{a:3,c:1}
Thanks @mattnibs!