Unable to insert data into table with field partition

[darshanmehta10]

There are 2 parts of this issue:

1. When a BigQuery insert fails, TableWriter class always assumes that the error is related to batch size and doesn't log it. E.g. this line needs to be present in the release candidates (currently it's only in master). While streaming the data, we got this exception from BigQuery:

Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
{
  "code" : 400,
  "errors" : [ {
    "domain" : "global",
    "message" : "Streaming to metadata partition of column based partitioning table <project>:<dataset>.<table>$20191031 is disallowed.",
    "reason" : "invalid"
  } ],
  "message" : "Streaming to metadata partition of column based partitioning table <project>:<dataset>.<table>$20191031 is disallowed.",
  "status" : "INVALID_ARGUMENT"
}

However, the component never logged that and instead, logged the following:

Attempted to reduce batch size below 1

after running out of splitting the batches.

2. It fails to insert into BigQuery when we use a field to partition the table (rather than legacy partitioning). It's happening because of this implementation. It always appends the partition decorator which eventually results in the below error:

Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
{
  "code" : 400,
  "errors" : [ {
    "domain" : "global",
    "message" : "Streaming to metadata partition of column based partitioning table <project>:<dataset>.<table>$20191031 is disallowed.",
    "reason" : "invalid"
  } ],
  "message" : "Streaming to metadata partition of column based partitioning table <project>:<dataset>.<table>$20191031 is disallowed.",
  "status" : "INVALID_ARGUMENT"
}

That logic needs to be conditional and controlled by a configuration parameter.

[archy-bold]

+1 Experiencing both these issues too.

[darshanmehta10]

@aarchy-bold This is fixed in PR https://github.com/wepay/kafka-connect-bigquery/pull/229