bigquery-emulator
bigquery-emulator copied to clipboard
goccy
fix(server): Support google.protobuf.Timestamp values for the storage write API
Summary
While BigQuery recommends to send TIMESTAMP columns encoded as int64, they also support google.protobuf.Timestamp: https://cloud.google.com/bigquery/docs/supported-data-types. This is critical for workloads that ingest JSON data, where RFC3339 strings can naturally decode into a timestamppb.Timestamp.
The client SDK making this easier with the following issue and PR:
- https://github.com/googleapis/google-cloud-go/issues/12569
- https://github.com/googleapis/google-cloud-go/pull/12579
Testing these changes work against Google, but panic the emulator:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x60 pc=0x10387d398]
goroutine 8764 [running]:
github.com/goccy/bigquery-emulator/types.normalizeData({0x10541b980?, 0x14002391318?}, 0x0)
/Users/coxley/projects/fork-bigquery-emulator/types/types.go:547 +0x88
github.com/goccy/bigquery-emulator/types.normalizeData({0x105529660?, 0x1400264d080?}, 0x140026da1e0)
/Users/coxley/projects/fork-bigquery-emulator/types/types.go:575 +0x394
github.com/goccy/bigquery-emulator/types.NewTableWithSchema(0x14002330248, {0x14001471238, 0x1, 0x103a9316c?})
/Users/coxley/projects/fork-bigquery-emulator/types/types.go:479 +0x398
github.com/goccy/bigquery-emulator/server.(*storageWriteServer).insertTableData(0x140025dcba0, {0x105869c28, 0x1400264d0b0}, 0x1400220c3d8, 0x1400083c150, {0x14001471238?, 0x4?, 0x10544dca0?})
/Users/coxley/projects/fork-bigquery-emulator/server/storage_handler.go:691 +0x44
github.com/goccy/bigquery-emulator/server.(*storageWriteServer).appendRows(0x140025dcba0, 0x1400083c2a0, {0x10588ca80, 0x14001612000}, {0x1058750d0, 0x14002430980})
/Users/coxley/projects/fork-bigquery-emulator/server/storage_handler.go:515 +0x428
github.com/goccy/bigquery-emulator/server.(*storageWriteServer).AppendRows(0x140025dcba0, {0x1058750d0, 0x14002430980})
/Users/coxley/projects/fork-bigquery-emulator/server/storage_handler.go:437 +0xb0
cloud.google.com/go/bigquery/storage/apiv1/storagepb._BigQueryWrite_AppendRows_Handler({0x105720fa0?, 0x140025dcba0}, {0x105870a20, 0x140026da3c0})
/Users/coxley/.go/pkg/mod/cloud.google.com/go/[email protected]/storage/apiv1/storagepb/storage.pb.go:3365 +0xdc
google.golang.org/grpc.(*Server).processStreamingRPC(0x140015d5000, {0x105869c28, 0x1400264cbd0}, {0x10587b040, 0x14000652d00}, 0x1400263e6c0, 0x140025dcc60, 0x106c72ac0, 0x0)
/Users/coxley/.go/pkg/mod/google.golang.org/[email protected]/server.go:1680 +0xe54
google.golang.org/grpc.(*Server).handleStream(0x140015d5000, {0x10587b040, 0x14000652d00}, 0x1400263e6c0)
/Users/coxley/.go/pkg/mod/google.golang.org/[email protected]/server.go:1801 +0xac4
google.golang.org/grpc.(*Server).serveStreams.func2.1()
/Users/coxley/.go/pkg/mod/google.golang.org/[email protected]/server.go:1027 +0x84
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 8622
/Users/coxley/.go/pkg/mod/google.golang.org/[email protected]/server.go:1038 +0x138
This PR ensures that google.protobuf.Timestamp messages are converted to time.Time early in decodeRowData.
It also fixes a bug for RECORD columns containing TIMESTAMP cells — but it depends on goccy/go-zetasqlite#232 to make the test pass.
Test Plan
Added a new test case that reproduces the problem, and is fixed with these changes:
> go test ./...
? github.com/goccy/bigquery-emulator/cmd/bigquery-emulator [no test files]
? github.com/goccy/bigquery-emulator/internal/connection [no test files]
? github.com/goccy/bigquery-emulator/internal/contentdata [no test files]
? github.com/goccy/bigquery-emulator/internal/logger [no test files]
? github.com/goccy/bigquery-emulator/internal/metadata [no test files]
? github.com/goccy/bigquery-emulator/internal/types [no test files]
ok github.com/goccy/bigquery-emulator/server 2.749s
? github.com/goccy/bigquery-emulator/types [no test files]
