target-bigquery
target-bigquery copied to clipboard
Feature: Create a config option to parse timestamps in special formats.
I have a timestamp coming from a source that causes BigQuery's default timestamp parsing to fail with how this package by default parses timestamps, this line of code generates this sql:
select
CAST('2023-04-26 14:57:00 -0700' as TIMESTAMP) as timestamp_without_space
Which fails for this reason:
The issue is that there's a space between the seconds and timezone offset, here's what comes from my source system2023-04-26 14:57:00 -0700
, however, here's what BigQuery's default CAST(.. as TIMESTAMP)
function wants 2023-04-26 14:57:00-0700
.
I would like to be able to add a config option to my meltano.yml
file for specifying how timestamps should be parsed, something like this:
- name: target-bigquery
variant: z3z1ma
pip_url: git+https://github.com/z3z1ma/target-bigquery.git
config:
credentials_json: ${BIGQUERY_CREDENTIALS_JSON}
project: some-project
dataset: some-dataset
timestamp_format: ''%Y-%m-%d %H:%M:%S %z'
and then the package would parse timestamps like this instead:
select
PARSE_TIMESTAMP('%Y-%m-%d %H:%M:%S %z', some_special_timestamp_column) AS parsed_timestamp