Support for configure BigQuery table partition type

[duanshiqiang]

The decision was made 2+ years ago to always use data-partitioned tables when letting kafka-connect-bigquery to manage BigQuery tables. (refer to https://github.com/wepay/kafka-connect-bigquery/issues/39#issuecomment-267175814)

But BigQuery now supports 2 different kinds of date-partitioned tables (the old partitioned by ingestion time type and the new field-based date-partitioned tables).

We are heavily using the field-based date-partitioned tables and currently how we integrate with kafka-connect-bigquery connector is that we manually create the tables required with field partition enabled and only using kafka-connect-bigquery to ingest data into them. But it is troublesome to do because we need to take care of schema update, etc.

It would be better to allow users to configure the table partition type. And I personally think that allowing users to disable partition totally should also be supported.

[duanshiqiang]

https://cloud.google.com/bigquery/docs/partitioned-tables

[magiciiboy]

@duanshiqiang totally agree and wait for this feature

[darshanmehta10]

@duanshiqiang @magiciiboy Added a PR for this: https://github.com/wepay/kafka-connect-bigquery/pull/229

[darrenhaken]

any update on this?