Vladyslav Bushynskyi
Vladyslav Bushynskyi
Bump. Also, this is our BigQueryIO config: ``` BigQueryIO .write() .to(dynamicDestination) .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED) .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND) .withoutValidation() .withFormatFunction(formatFunction) .optimizedWrites() .withFailedInsertRetryPolicy(InsertRetryPolicy.retryTransientErrors()) .withSchemaUpdateOptions( setOf( BigQueryIO.Write.SchemaUpdateOption.ALLOW_FIELD_ADDITION, BigQueryIO.Write.SchemaUpdateOption.ALLOW_FIELD_RELAXATION ) ) .withMethod(BigQueryIO.Write.Method.STORAGE_WRITE_API) ```
Thanks for a great write-up. We have experienced the same thing in our production services as well, when there is a big rate of messages coming, some tend to get...
I have since tried to `DELETE` the resumable upload by `crc32` when I destroy my write/validate streams but it also doesn't affect the underlying streams, so the handle stays open.
Also, it seems like actual BigQuery returns null on NULLABLE fields with no value, while the emulator does not return anything, which seems wrong.