scio icon indicating copy to clipboard operation
scio copied to clipboard

Could we add `SchemaUpdateOption` param to BQ WriteParam?

Open arthurenc opened this issue 5 months ago • 1 comments

CONTEXT

We are trying to use saveAsBigQueryTable to append data to a table where additional fields might be added to the schema. At the moment, we are unable to do so as the schema for the data we want to append to the table has additional fields not present in the table schema.

"errors" : [ {
      "message" : "Provided Schema does not match Table table_name. Cannot add fields (field: field_name)",
      "reason" : "invalid"
    }

SOLUTION

Google offers a parameter SchemaUpdateOptions which allows you to update the schema of a destination table (docs). This parameter can be configured in two ways; ALLOW_FIELD_ADDITION and ALLOW_FIELD_RELAXATION. If we configure it to ALLOW_FIELD_ADDITION then additional nullable fields, that are not already present in the table schema, could be appended using the saveAsBigQueryTable method.

At the moment this is not offered as a configuration option with the scio wrapper. Would it be possible to get this added in?

arthurenc avatar Jan 15 '24 15:01 arthurenc