kafka-connect-bigquery
kafka-connect-bigquery copied to clipboard
BigQueryConnectException should log offsets for failed rows
Currently when a BigQueryConnectException
is thrown, the formatInsertAllErrors
printer outputs some information about which rows failed and with what error. Unfortunately the end user cannot correlate these rows at all back to the actual message that failed to insert as I see it. We should propagate the offset information to this exception if possible and print it when this exception is thrown so that users get actionable information. From my understanding this would entail modifying the BigQueryWriter
to work with lists of Tuple[Offset, InsertAllRequest.RowToInsert] or some similar structure instead of just the rows, but I would appreciate guidance on how to go about this.