bigquery-emulator icon indicating copy to clipboard operation
bigquery-emulator copied to clipboard

Several issues when using with Node.js Bigquery client

Open vbshnsk opened this issue 2 years ago • 1 comments

Hey! Thank you for doing such an excellent job on this emulator! We are currently trying to migrate our test environment to the emulator. While it is partly blocked by another issue I left on your zetasql implementation, we're also experiencing some troubles with usage with Node.js client library.

  1. Inserting empty arrays to record fields and retrieving them after results in error.

Assume you have the following schema:

 {
   name: 'someArrayField',
   mode: 'REPEATED',
   type: 'STRING',
 }

Then you try to insert an empty array using node client, querying it afterwards:

await dataset.table('table').insert([{ someArrayField: [] }])
await dataset.query(
   `SELECT * FROM 'table'`,
);

This results in an error in the Node client lib as the emulator returns nothing for the array field:

bigquery_1  | 2023-01-23T13:15:10.609Z  INFO    contentdata/repository.go:138           {"query": "SELECT * FROM test1.example1", "values": []}
bigquery_1  | 2023-01-23T13:15:10.610Z  DEBUG   contentdata/repository.go:209   query result    {"rows": [[[]]]}
            return row.f.map((field, index) => {
                         ^
TypeError: Cannot read properties of null (reading 'map')
    at /Users/vbshnsk/Desktop/extenda/hiiretail-txr-search-api/node_modules/@google-cloud/bigquery/build/src/bigquery.js:290:35
    at Array.map (<anonymous>)
    at mergeSchema (/Users/vbshnsk/Desktop/extenda/hiiretail-txr-search-api/node_modules/@google-cloud/bigquery/build/src/bigquery.js:286:26)
    at Array.map (<anonymous>)
    at Function.mergeSchemaWithRows_ (/Users/vbshnsk/Desktop/extenda/hiiretail-txr-search-api/node_modules/@google-cloud/bigquery/build/src/bigquery.js:283:14)
    at /Users/vbshnsk/Desktop/extenda/hiiretail-txr-search-api/node_modules/@google-cloud/bigquery/build/src/job.js:365:44
    at /Users/vbshnsk/Desktop/extenda/hiiretail-txr-search-api/node_modules/@google-cloud/common/build/src/util.js:412:25
    at Util.handleResp (/Users/vbshnsk/Desktop/extenda/hiiretail-txr-search-api/node_modules/@google-cloud/common/build/src/util.js:161:9)
    at /Users/vbshnsk/Desktop/extenda/hiiretail-txr-search-api/node_modules/@google-cloud/common/build/src/util.js:534:22
    at onResponse (/Users/vbshnsk/Desktop/extenda/hiiretail-txr-search-api/node_modules/retry-request/index.js:240:7)
  1. Sending number as a parameter value in query fails

Assume you have the following schema:

 {
   name: 'someNumberField',
   mode: 'NULLABLE',
   type: 'INT64',
 }

Inserting data:

await dataset.table('table').insert([{ someNumberField: 123 }])

Selecting with param

await dataset.query(
   `SELECT * FROM 'table' WHERE someNumberField = @someNumberField`,
   { params: { someNumberField: 123 } }
);

This results in a Bad Request error and following logs:

bigquery_1  | 2023-01-23T13:22:32.165Z  ERROR   server/handler.go:1007  invalid {"error": "invalid: json: cannot unmarshal number into Go struct field QueryParameterValue.Value of type string"}
  1. Cannot send array parameters

Assume you have the following schema:

 {
   name: 'someStringField',
   mode: 'NULLABLE',
   type: 'STRING',
 }

Inserting data:

await dataset.table('table').insert([{ someStringField: "123" }])

Selecting with param

await dataset.query(
   `SELECT * FROM 'table' WHERE someStringField = IN UNNEST(@someStringField)`,
   { params: { someStringField: ["123"]},  types: { someStringField: ['STRING']} }
);

Results in following log:

bigquery_1  | 2023-01-23T13:36:12.121Z  INFO    contentdata/repository.go:138           {"query": "SELECT * FROM test1.example1 WHERE someStringField = IN UNNEST(@someStringField)", "values": [{"Name":"someStringField","Value":""}]}

And error:

failed to parse statements: failed to parse statement: INVALID_ARGUMENT: Syntax error: Unexpected keyword IN [type.googleapis.com/zetasql.ErrorLocation='\x08\x01\x10\x36']

Node client sends this param with QueryParameter.arrayValues[]: https://cloud.google.com/bigquery/docs/reference/rest/v2/QueryParameter#queryparametervalue

which is not handled in the emulator it seems

vbshnsk avatar Jan 23 '23 13:01 vbshnsk

Also, it seems like actual BigQuery returns null on NULLABLE fields with no value, while the emulator does not return anything, which seems wrong.

vbshnsk avatar Jan 24 '23 10:01 vbshnsk