Trouble Integrating BigQuery Emulator with Write Streams Java Library : runtime error: invalid memory address or nil pointer dereference

[maroodb]

Hi,

I'm trying to make the BigQuery emulator work with the Write Streams Java library, but it's not cooperating. The container is logging an error. Check out the code snippet below:

ManagedChannel channel = ManagedChannelBuilder.forTarget(url).usePlaintext().build();
    TransportChannelProvider channelProvider = FixedTransportChannelProvider.create(
        GrpcTransportChannel.create(channel)
    );

    var client =BigQueryWriteSettings.Builder bigQueryWriteSettings = BigQueryWriteSettings.newBuilder()
        .setEndpoint(url)
        .setTransportChannelProvider(channelProvider)
        .setQuotaProjectId("test-project")
        .setHeaderProvider(new UserAgentHeaderProvider(GcpBigQueryAutoConfiguration.class))
        .setCredentialsProvider(new NoCredentialsProvider())
       .build();


 var jsonStream = JsonStreamWriter jsonStreamWriter = JsonStreamWriter.newBuilder(writeStream.getName(),
              writeStream.getTableSchema())
          .setEndpoint(client.getSettings().getEndpoint())
          .setCredentialsProvider(client.getSettings().getCredentialsProvider())
          .setChannelProvider(client.getSettings().getTransportChannelProvider())
          .build()

var jsonArray = someData();

jsonStream.append(jsonArray);

The container logs

2023-11-16 09:39:32 [bigquery-emulator] REST server listening at 0.0.0.0:54292
2023-11-16 09:39:32 [bigquery-emulator] gRPC server listening at 0.0.0.0:54293
2023-11-16 09:47:32 panic: runtime error: invalid memory address or nil pointer dereference
2023-11-16 09:47:32 [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x17016b1]
2023-11-16 09:47:32 
2023-11-16 09:47:32 goroutine 111 [running]:
2023-11-16 09:47:32 github.com/goccy/bigquery-emulator/server.(*storageWriteServer).appendRows(0xc0001bbdd0, 0xc0002425b0, {0x31a0048, 0xc000239ad0}, {0x318eb38, 0xc00070e2a0})
2023-11-16 09:47:32     /work/server/storage_handler.go:491 +0x211
2023-11-16 09:47:32 github.com/goccy/bigquery-emulator/server.(*storageWriteServer).AppendRows(0xc000a4ab10?, {0x318eb38, 0xc00070e2a0})
2023-11-16 09:47:32     /work/server/storage_handler.go:438 +0xd4
2023-11-16 09:47:32 cloud.google.com/go/bigquery/storage/apiv1/storagepb._BigQueryWrite_AppendRows_Handler({0x2d74000?, 0xc0001bbdd0}, {0x318bc98?, 0xc0002354a0})
2023-11-16 09:47:32     /go/pkg/mod/cloud.google.com/go/[email protected]/storage/apiv1/storagepb/storage.pb.go:3250 +0x9f
2023-11-16 09:47:32 google.golang.org/grpc.(*Server).processStreamingRPC(0xc0000001e0, {0x31946e0, 0xc00081e340}, 0xc000254000, 0xc0001bbe90, 0x4aba2a0, 0x0)
2023-11-16 09:47:32     /go/pkg/mod/google.golang.org/[email protected]/server.go:1639 +0x1277
2023-11-16 09:47:32 google.golang.org/grpc.(*Server).handleStream(0xc0000001e0, {0x31946e0, 0xc00081e340}, 0xc000254000, 0x0)
2023-11-16 09:47:32     /go/pkg/mod/google.golang.org/[email protected]/server.go:1726 +0x9d6
2023-11-16 09:47:32 google.golang.org/grpc.(*Server).serveStreams.func1.2()
2023-11-16 09:47:32     /go/pkg/mod/google.golang.org/[email protected]/server.go:966 +0x98
2023-11-16 09:47:32 created by google.golang.org/grpc.(*Server).serveStreams.func1
2023-11-16 09:47:32     /go/pkg/mod/google.golang.org/[email protected]/server.go:964 +0x28a

The same code works on with a real BigQuery with a DefaultCredentialProvider.

[totem3]

Thank you for your report. To better understand and address the issue, could you please provide detailed steps to reproduce it, or executable code if possible?

[AbhinavAtul]

+1

[seanmcnealy]

The Java client doesn't supply an offset. At least for the first message I'm not sure about later.

https://github.com/goccy/bigquery-emulator/pull/261 is one quick way to get this working for Java. Though there may be a better answer.