protobuf icon indicating copy to clipboard operation
protobuf copied to clipboard

Support for schema on the fly to unlock bigquery Write API

Open psarka opened this issue 8 months ago • 0 comments

This is probably a duplicate of https://github.com/eigenein/protobuf/issues/140, but with an interesting twist.

Bigquery (a gcloud data warehouse) has a WriteAPI, that is a major pain to use. Not only you need to send your data in protobuf format, but you also need to advertise your payload schema in protobuf :-/

Documentation of this capability is terrible, but this example is a good illustration on how it works:

https://github.com/xebia/bigquery-storage-write-api-examples/blob/main/src/bigquery_storage_write_api_examples/examples/pending_type_stream_writer_example.py

If this library supported exporting schema in protobuf, it would be a major improvement and would essentially unlock the write API for normal python users.

psarka avatar Apr 11 '25 17:04 psarka