transfer icon indicating copy to clipboard operation
transfer copied to clipboard

Published 20 hours ago • verified publisher icon artie-labs

Use BigQuery Storage Write API

Open Tang8330 opened this issue 2 years ago • 2 comments

https://cloud.google.com/bigquery/docs/write-api

Context https://artie-technologies.slack.com/archives/C04Q7GG6DT9/p1693997817550579

Tang8330 avatar Sep 06 '23 14:09 Tang8330

Potentially doable with the dynamic JSON parser: https://github.com/googleapis/google-cloud-go/blob/7a46b5428f239871993d66be2c7c667121f60a6f/bigquery/storage/managedwriter/integration_test.go#L424-L442

Tang8330 avatar Feb 17 '24 19:02 Tang8330

FYI, https://cloud.google.com/pubsub/docs/bigquery#change_data_capture

This looks like a way simpler alternative to interfacing with Storage Write directly. Pub/Sub supports a native BigQuery subscription, which uses Storage Write under the hood.

Transfer could simply register those native BigQuery subscriptions to the topics it already registers in Pub/Sub and potentially replace the temp table / merge query workflow altogether.

claytongentry avatar May 04 '24 22:05 claytongentry

Completed: https://github.com/artie-labs/transfer/pull/715

Tang8330 avatar Jun 14 '24 05:06 Tang8330