scio
scio copied to clipboard
Could we add `SchemaUpdateOption` param to BQ WriteParam?
CONTEXT
We are trying to use saveAsBigQueryTable to append data to a table where additional fields might be added to the schema. At the moment, we are unable to do so as the schema for the data we want to append to the table has additional fields not present in the table schema.
"errors" : [ {
"message" : "Provided Schema does not match Table table_name. Cannot add fields (field: field_name)",
"reason" : "invalid"
}
SOLUTION
Google offers a parameter SchemaUpdateOptions
which allows you to update the schema of a destination table (docs). This parameter can be configured in two ways; ALLOW_FIELD_ADDITION and ALLOW_FIELD_RELAXATION. If we configure it to ALLOW_FIELD_ADDITION then additional nullable fields, that are not already present in the table schema, could be appended using the saveAsBigQueryTable method.
At the moment this is not offered as a configuration option with the scio wrapper. Would it be possible to get this added in?