fluent-plugin-bigquery icon indicating copy to clipboard operation
fluent-plugin-bigquery copied to clipboard

Published 20 hours ago • verified publisher icon fluent-plugins-nursery

Add new feature of bigquery-avro-load job

Open seii-saintway opened this issue 5 years ago • 3 comments

Work together with fluent-plugin-avro

Acknowledgements

I would like to extend my sincere gratitude to my team leader, for his instructive advice and useful discussions make me realize where the problem is.

seii-saintway avatar Jun 11 '19 01:06 seii-saintway

There’re API errors of EOF, because of the synchronization problem of the variable @avro_writer.block_count. And I feel difficult to solve this problem, so maybe it’s better to solve the avro-format problem in the fluent-plugin-avro.

seii-saintway avatar Jun 16 '19 03:06 seii-saintway

=> Use chunk.size to reset @avro_writer.block_count

seii-saintway avatar Jun 18 '19 07:06 seii-saintway

I found a mistaken in README.md : chunk_records_limit → chunk_limit_records

seii-saintway avatar Jun 18 '19 08:06 seii-saintway

I'm sorry. This method does not seem to have any performance advantages. It would be better to generate avro with a formatter plugin.

joker1007 avatar Oct 05 '22 13:10 joker1007