parca-dev
Panic after startup in current Parca on `main`
After updating Parca to main-bd028ff3 on demo.parca.dev it panics after a few seconds with the following. It's probably something to fix in FrostDB first.
panic: runtime error: index out of range [0] with length 0
goroutine 2012 [running]:
github.com/polarsignals/frostdb/pqarrow.recordBuilderLength(0xe4a627?)
github.com/polarsignals/[email protected]/pqarrow/arrow.go:451 +0xef
github.com/polarsignals/frostdb/pqarrow.distinctColumnsToArrowRecord({0x4a28ab0, 0xc006ed2ed0}, {0x4a1a010, 0x67e22e8}, 0xc006ee8640, {0xc000d048c0, 0x13, 0x7f0b01fa4cf0?}, {0xc001220280, 0x13, ...}, ...)
github.com/polarsignals/[email protected]/pqarrow/arrow.go:475 +0x71
github.com/polarsignals/frostdb/pqarrow.contiguousParquetRowGroupToArrowRecord({0x4a28ab0, 0xc006ed2ed0}, {0x4a1a010, 0x67e22e8}, {0x7f0b01f544d8, 0xc006ed2f30}, 0xc006ee8640, {0x0, 0x0?}, {0xc000fa5470, ...}, ...)
github.com/polarsignals/[email protected]/pqarrow/arrow.go:241 +0x1fe
github.com/polarsignals/frostdb/pqarrow.ParquetRowGroupToArrowRecord({0x4a28ab0?, 0xc006ed2ed0?}, {0x4a1a010?, 0x67e22e8?}, {0x7f0b01f544d8?, 0xc006ed2f30?}, 0x1970a200705a808?, {0x0?, 0x0?}, {0xc000fa5470, ...}, ...)
github.com/polarsignals/[email protected]/pqarrow/arrow.go:115 +0xbe
github.com/polarsignals/frostdb.(*Table).Iterator(0xc0001e22d0, {0x4a28ab0?, 0xc006ed2d50?}, 0xc000100000?, {0x4a1a010, 0x67e22e8}, 0xc006ee8640, {{0xc000fa5460, 0x1, 0x1}, ...}, ...)
github.com/polarsignals/[email protected]/table.go:524 +0xa91
github.com/polarsignals/frostdb/query/physicalplan.(*TableScan).Execute.func1({0x4a28ab0, 0xc006ed2d50}, 0xc006ed2cf0?)
github.com/polarsignals/[email protected]/query/physicalplan/physicalplan.go:126 +0x351
github.com/polarsignals/frostdb.(*Table).View(0xc0001e22d0, {0x4a28ab0?, 0xc006ed2cf0?}, 0xc006ed2d20)
github.com/polarsignals/[email protected]/table.go:441 +0x28c
github.com/polarsignals/frostdb/query/physicalplan.(*TableScan).Execute(0xc006ed2cc0, {0x4a28ab0?, 0xc006ed2c30?}, {0x4a1a010?, 0x67e22e8})
github.com/polarsignals/[email protected]/query/physicalplan/physicalplan.go:110 +0x1cc
github.com/polarsignals/frostdb/query/physicalplan.(*OutputPlan).Execute(...)
github.com/polarsignals/[email protected]/query/physicalplan/physicalplan.go:83
github.com/polarsignals/frostdb/query.LocalQueryBuilder.Execute({{0x4a1a010, 0x67e22e8}, {0x4a07920, 0x67e22e8}, {0xc0010fcd40}}, {0x4a28ab0?, 0xc006ed2b40?}, 0xc000fa5430)
github.com/polarsignals/[email protected]/query/engine.go:141 +0x242
github.com/parca-dev/parca/pkg/parcacol.(*Querier).Values(0xc000fbeb40, {0x4a28ab0, 0xc006ed2b40}, {0xc000054ea0, 0x19}, {0xc00a60b388?, 0x2d3f391?, 0x3a904e0?}, {0x14464d56, 0xedaabe06f, ...}, ...)
github.com/parca-dev/parca/pkg/parcacol/querier.go:129 +0x1ab
github.com/parca-dev/parca/pkg/query.(*ColumnQueryAPI).Values(0xc000fbeb80, {0x4a28ab0, 0xc006ed2b40}, 0xc0070cb740)
github.com/parca-dev/parca/pkg/query/columnquery.go:81 +0x10e
github.com/parca-dev/parca/gen/proto/go/parca/query/v1alpha1._QueryService_Values_Handler.func1({0x4a28ab0, 0xc006ed2b40}, {0x38b0a80?, 0xc0070cb740})
github.com/parca-dev/parca/gen/proto/go/parca/query/v1alpha1/query_vtproto.pb.go:287 +0x78
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.UnaryServerInterceptor.func1({0x4a28ab0, 0xc006ed2b40}, {0x38b0a80, 0xc0070cb740}, 0x0?, 0xc006ec0060)
github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:22 +0x21e
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4a28ab0?, 0xc006ed2b40?}, {0x38b0a80?, 0xc0070cb740?})
github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1({0x4a28ab0, 0xc006ed2b40}, {0x38b0a80, 0xc0070cb740}, 0x0?, 0xc006fd44a0)
github.com/grpc-ecosystem/[email protected]/server_metrics.go:107 +0x87
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4a28ab0?, 0xc006ed2b40?}, {0x38b0a80?, 0xc0070cb740?})
github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x4a28ab0, 0xc006ed2a50}, {0x38b0a80, 0xc0070cb740}, 0xc006fd4480, 0xc006fd44c0)
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:325 +0x676
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4a28ab0?, 0xc006ed2a50?}, {0x38b0a80?, 0xc0070cb740?})
github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x4a28ab0, 0xc006ed2a50}, {0x38b0a80, 0xc0070cb740}, 0xc006d19ad0?, 0x3377580?)
github.com/grpc-ecosystem/[email protected]/chain.go:34 +0xbe
github.com/parca-dev/parca/gen/proto/go/parca/query/v1alpha1._QueryService_Values_Handler({0x38780c0?, 0xc000fbeb80}, {0x4a28ab0, 0xc006ed2a50}, 0xc000340a80, 0xc00121ea50)
github.com/parca-dev/parca/gen/proto/go/parca/query/v1alpha1/query_vtproto.pb.go:289 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000f521e0, {0x4a3ba20, 0xc006ffecf0}, 0xc000d80900, 0xc00121edb0, 0x6780518, 0x0)
google.golang.org/[email protected]/server.go:1301 +0xb2b
google.golang.org/grpc.(*Server).handleStream(0xc000f521e0, {0x4a3ba20, 0xc006ffecf0}, 0xc000d80900, 0x0)
google.golang.org/[email protected]/server.go:1642 +0xa2f
google.golang.org/grpc.(*Server).serveStreams.func1.2()
google.golang.org/[email protected]/server.go:938 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
google.golang.org/[email protected]/server.go:936 +0x28a
```
the line that panics: https://github.com/polarsignals/frostdb/blob/beada053c7d916e9681b93905eaa49e423af1b88/pqarrow/arrow.go#L451
~The upstream fix is here: https://github.com/segmentio/parquet-go/pull/337~ Sorry, that was not correct.
My intuition is that for some reason https://github.com/polarsignals/frostdb/blob/beada053c7d916e9681b93905eaa49e423af1b88/query/physicalplan/physicalplan.go#L111 is returning a non-nil zero-length schema. I'm not sure how this is possible since we iterate over all the row groups to construct the arrow schema. If the table were empty, then we shouldn't be iterating over any row groups in Iterator. Is there a way to simplify the repro?
In (https://github.com/polarsignals/frostdb/pull/195), I added a protection for a zero-length schema so that we don't panic and instead return an error. We should still figure out why this is happening.
Added a fix to https://github.com/polarsignals/frostdb/pull/195 for the old behavior of emitting zero rows in the case of an empty schema. I now see how it can happen (empty table, context cancellation, or filter), but I'm not sure I like it. Once that PR is in, this panic shouldn't happen anymore.