kafka-connect-bigquery icon indicating copy to clipboard operation
kafka-connect-bigquery copied to clipboard

BigQueryConnectException should log offsets for failed rows

Open muscovitebob opened this issue 4 years ago • 0 comments

Currently when a BigQueryConnectException is thrown, the formatInsertAllErrors printer outputs some information about which rows failed and with what error. Unfortunately the end user cannot correlate these rows at all back to the actual message that failed to insert as I see it. We should propagate the offset information to this exception if possible and print it when this exception is thrown so that users get actionable information. From my understanding this would entail modifying the BigQueryWriter to work with lists of Tuple[Offset, InsertAllRequest.RowToInsert] or some similar structure instead of just the rows, but I would appreciate guidance on how to go about this.

muscovitebob avatar Jun 07 '20 11:06 muscovitebob