java.lang.NullPointerException: Cannot invoke "com.google.cloud.bigquery.TableId.getProject()" because "tableId" is null

[pavelfomin]

What happened?

The following code

        QueryJobConfiguration queryJobConfiguration = QueryJobConfiguration
            .newBuilder("SELECT * FROM dataset.table")
            .build()

        JobId jobId = JobId.newBuilder().setProject(project).setRandomJob().build()
        Job job = bigQuery.create(JobInfo.newBuilder(queryJobConfiguration)
            .setJobId(jobId)
            .build())

        TableResult tableResult = job.getQueryResults()

fails with

Cannot invoke "com.google.cloud.bigquery.TableId.getProject()" because "tableId" is null
java.lang.NullPointerException: Cannot invoke "com.google.cloud.bigquery.TableId.getProject()" because "tableId" is null
	at com.google.cloud.bigquery.BigQueryImpl.listTableData(BigQueryImpl.java:1704)
	at com.google.cloud.bigquery.BigQueryImpl.listTableData(BigQueryImpl.java:1671)
	at com.google.cloud.bigquery.Job.getQueryResults(Job.java:429)

See https://github.com/pavelfomin/spring-boot-bq-example/blob/main/src/test/groovy/com/droidablebee/springboot/bq/service/BigQueryServiceISpec.groovy. The first scenario fails when destinationTable is not set (works with real BigQuery). The second scenario sets the destinationTable explicitly as a workaround.

What did you expect to happen?

In real BigQuery:

When I run a query, BigQuery creates a temporary destination table to store the results. Job.getQueryResults() fetches rows from this temporary table, regardless of the original table schema. It also sets QueryJobConfiguration.destinationTable with the reference to that temporary table.

Why it works:

BigQuery dynamically creates the destination table schema based on the query output, not the source table. The client library fetches results from this destination table, which matches the query output.

Why the emulator fails:

The emulator does not fully implement this behavior. It may not create a destination table with the correct schema, or may not set the destinationTable field in the job metadata. The Java client expects this metadata to be present and correct, so it can fetch results from the temporary table. If the schema doesn’t match, or the destination table is missing, you get errors (like NullPointerException).

How can we reproduce it (as minimally and precisely as possible)?

git clone [email protected]:pavelfomin/spring-boot-bq-example.git
./gradlew clean build

Anything else we need to know?

No response

[ohaibbq]

Hi @pavelfomin! This looks like a duplicate of #183. We recently merged and released a fix to our fork (Recidiviz/bigquery-emulator#35).

Do you want to try the image ghcr.io/recidiviz/bigquery-emulator:0.4.4-recidiviz.26? We will be releasing a new version that is up to date with upstream in the next few days.

[pavelfomin]

When I tried using "ghcr.io/recidiviz/bigquery-emulator:0.4.4-recidiviz.26" to instantiate BigQueryEmulatorContainer

BigQueryEmulatorContainer container = new BigQueryEmulatorContainer("ghcr.io/recidiviz/bigquery-emulator:0.4.4-recidiviz.26")

I got this init error from test containers:

Caused by: java.lang.IllegalStateException: Failed to verify that image 'ghcr.io/recidiviz/bigquery-emulator:0.4.4-recidiviz.26' is a compatible substitute for 'ghcr.io/goccy/bigquery-emulator'. This generally means that you are trying to use an image that Testcontainers has not been designed to use. If this is deliberate, and if you are confident that the image is compatible, you should declare compatibility in code using the `asCompatibleSubstituteFor` method. For example:
   DockerImageName myImage = DockerImageName.parse("ghcr.io/recidiviz/bigquery-emulator:0.4.4-recidiviz.26").asCompatibleSubstituteFor("ghcr.io/goccy/bigquery-emulator");
and then use `myImage` instead.
	at org.testcontainers.utility.DockerImageName.assertCompatibleWith(DockerImageName.java:279)
	at org.testcontainers.containers.BigQueryEmulatorContainer.<init>(BigQueryEmulatorContainer.java:27)
	at org.testcontainers.containers.BigQueryEmulatorContainer.<init>(BigQueryEmulatorContainer.java:22)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at com.droidablebee.springboot.bq.BaseIntegrationSpec.<clinit>(BaseIntegrationSpec.groovy:19)

[simi]

@pavelfomin can you try as described in error message, something like following?

DockerImageName myImage = DockerImageName.parse("ghcr.io/recidiviz/bigquery-emulator:0.4.4-recidiviz.26").asCompatibleSubstituteFor("ghcr.io/goccy/bigquery-emulator");
BigQueryEmulatorContainer container = new BigQueryEmulatorContainer(myImage);

[pavelfomin]

That worked, thank you for pointing this out.

I also verified that running application locally with docker-compose using ghcr.io/recidiviz/bigquery-emulator:0.4.4-recidiviz.26 works as expected.

Thank you for making this fix. Looking forward to using the new release version soon!

[pavelfomin]

Looks like this fix has also been included in ghcr.io/recidiviz/bigquery-emulator:0.6.6-recidiviz.1