scio icon indicating copy to clipboard operation
scio copied to clipboard

Support BigQuery Load jobs for batch loading

Open RustedBones opened this issue 2 years ago • 4 comments

From beam documentation

When you specify load jobs as the insertion method using BigQueryIO.write().withMethod(FILE_LOADS).

Scio should also give users the possibility to use files to load data to BigQuery

RustedBones avatar Apr 27 '22 12:04 RustedBones

@RustedBones -- not sure I understand your issue here.

Is there something you are trying to do that you are not able to ( asking about functionality, not specific syntax/methods ).

brucearctor avatar Apr 28 '22 16:04 brucearctor

Ah, based on your other issue ( https://github.com/spotify/scio/issues/4333 ), it looks like you're thinking Scio would benefit from BeamIO.Write Methods --> https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.Method.html ?

brucearctor avatar Apr 28 '22 17:04 brucearctor

That's correct. At the moment those are not exposed in the Scio API. Both tickets have been opened after user's request, feel free to assign yourself if you want to work on them.

RustedBones avatar Apr 29 '22 11:04 RustedBones

I will need to dig into specifics a bit more -- as don't want to assign myself, until sure that have the ability to see it through relatively quickly ( to not be a holdup/blocker )

brucearctor avatar Apr 29 '22 13:04 brucearctor