Read storage API error selecting array field

[vladDotH]

What happened?

When i try to read field with <Array> type via grpc, emulator produces error

What did you expect to happen?

Successful quiering including array fields

How can we reproduce it (as minimally and precisely as possible)?

code, where i create table with array and read it via grpc service: https://gist.github.com/vladDotH/1571031a34a384248870ff8bbb82f2ae

docker-compose.yml:

services:
  bq-emulator:
    container_name: bq-emulator
    image: ghcr.io/goccy/bigquery-emulator:0.6.6
    command: --project=testproject --database=testdatabase --dataset=testdataset --log-level=debug 
    ports:
      - 9050:9050
      - 9060:9060

Anything else we need to know?

I suppose it works correct, because i see my data in logs, but something goes wrong at arrow serializing stage

Stacktrace:

bq-emulator | [bigquery-emulator] REST server listening at 0.0.0.0:9050 bq-emulator | [bigquery-emulator] gRPC server listening at 0.0.0.0:9060 bq-emulator | 2025-04-09T20:42:53.008Z INFO server/middleware.go:63 POST /projects/testproject/jobs {"query": "alt=json&prettyPrint=false"} bq-emulator | 2025-04-09T20:42:53.013Z INFO contentdata/repository.go:166 {"query": "\n\t\tcreate table if not exists testdataset.testtable(\n\t\t\tx int64,\n\t\t\ty string,\n\t\t\ta Array\n\t\t);\n\n\t\tinsert into testdataset.testtable(x, y, a)\n\t\tvalues \n\t\t(1, '1 str', [1,2,3]), \n\t\t(2, '2nd str', []), \n\t\t(33, '3rd string', [0,0,0]);\n\t", "values": []} bq-emulator | 2025-04-09T20:42:53.018Z DEBUG contentdata/repository.go:245 query result {"rows": []} bq-emulator | 2025-04-09T20:42:53.041Z INFO server/middleware.go:63 GET /projects/testproject/queries/zvmrIEDeyc5yn7s1ez2Y5C6neIQ {"query": "alt=json&formatOptions.useInt64Timestamp=true&location=&maxResults=0&prettyPrint=false"} bq-emulator | 2025-04-09T20:42:53.145Z INFO server/middleware.go:63 GET /projects/testproject/jobs/zvmrIEDeyc5yn7s1ez2Y5C6neIQ {"query": "alt=json&fields=status%2Cstatistics&prettyPrint=false"} bq-emulator | 2025-04-09T20:42:53.148Z INFO server/middleware.go:63 POST /projects/testproject/queries {"query": "alt=json&prettyPrint=false"} bq-emulator | 2025-04-09T20:42:53.151Z INFO contentdata/repository.go:166 {"query": "select * from testdataset.testtable;", "values": []} bq-emulator | 2025-04-09T20:42:53.152Z DEBUG contentdata/repository.go:245 query result {"rows": [[1,"1 str",[1,2,3]],[2,"2nd str",[]],[33,"3rd string",[0,0,0]]]} bq-emulator | 2025-04-09T20:42:53.159Z INFO contentdata/repository.go:166 {"query": "SELECT x,y,a FROM testtable ", "values": []} bq-emulator | 2025-04-09T20:42:53.160Z DEBUG contentdata/repository.go:245 query result {"rows": [[1,"1 str",[1,2,3]],[2,"2nd str",[]],[33,"3rd string",[0,0,0]]]} bq-emulator | panic: arrow/array: field 2 has 6 rows. want=3 bq-emulator | bq-emulator | goroutine 335 [running]: bq-emulator | github.com/apache/arrow/go/v10/arrow/array.(*RecordBuilder).NewRecord(0xc000b9b8c0) bq-emulator | /go/pkg/mod/github.com/apache/arrow/go/[email protected]/arrow/array/record.go:321 +0x28e bq-emulator | github.com/goccy/bigquery-emulator/server.(*storageReadServer).sendARROWRows(0xc000c839b0?, 0xc000f13b80, 0xc00093be50, {0x30fb580, 0xc000d812a0}) bq-emulator | /work/server/storage_handler.go:339 +0x1a5 bq-emulator | github.com/goccy/bigquery-emulator/server.(*storageReadServer).ReadRows(0xc000c839b0, 0xc000b9af40, {0x30fb580, 0xc000d812a0}) bq-emulator | /work/server/storage_handler.go:166 +0x11c bq-emulator | cloud.google.com/go/bigquery/storage/apiv1/storagepb._BigQueryRead_ReadRows_Handler({0x2ce8c20?, 0xc000c839b0}, {0x30f9698, 0xc000ac03c0}) bq-emulator | /go/pkg/mod/cloud.google.com/go/[email protected]/storage/apiv1/storagepb/storage.pb.go:3048 +0xd0 bq-emulator | google.golang.org/grpc.(*Server).processStreamingRPC(0xc000153600, {0x30f44e8, 0xc000d85fb0}, {0x31034e0, 0xc000b73860}, 0xc000af6fc0, 0xc000c83a70, 0x4b68380, 0x0) bq-emulator | /go/pkg/mod/google.golang.org/[email protected]/server.go:1680 +0x1288 bq-emulator | google.golang.org/grpc.(*Server).handleStream(0xc000153600, {0x31034e0, 0xc000b73860}, 0xc000af6fc0) bq-emulator | /go/pkg/mod/google.golang.org/[email protected]/server.go:1801 +0xfbb bq-emulator | google.golang.org/grpc.(*Server).serveStreams.func2.1() bq-emulator | /go/pkg/mod/google.golang.org/[email protected]/server.go:1027 +0x8b bq-emulator | created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 338 bq-emulator | /go/pkg/mod/google.golang.org/[email protected]/server.go:1038 +0x135 bq-emulator exited with code 2

[ohaibbq]

I've implemented and released a fix for the Arrow array serialization panic in the Recidiviz fork of the emulator https://github.com/Recidiviz/bigquery-emulator/releases/tag/v0.6.6-recidiviz.0