target-bigquery icon indicating copy to clipboard operation
target-bigquery copied to clipboard

Feature: Create a config option to parse timestamps in special formats.

Open jcbmllgn opened this issue 1 year ago • 1 comments

I have a timestamp coming from a source that causes BigQuery's default timestamp parsing to fail with how this package by default parses timestamps, this line of code generates this sql:

select 
    CAST('2023-04-26 14:57:00 -0700' as TIMESTAMP) as timestamp_without_space

Which fails for this reason:

image

The issue is that there's a space between the seconds and timezone offset, here's what comes from my source system2023-04-26 14:57:00 -0700, however, here's what BigQuery's default CAST(.. as TIMESTAMP) function wants 2023-04-26 14:57:00-0700.


I would like to be able to add a config option to my meltano.yml file for specifying how timestamps should be parsed, something like this:

  - name: target-bigquery
    variant: z3z1ma
    pip_url: git+https://github.com/z3z1ma/target-bigquery.git
    config:
      credentials_json: ${BIGQUERY_CREDENTIALS_JSON}
      project: some-project
      dataset: some-dataset
      timestamp_format: ''%Y-%m-%d %H:%M:%S %z'

and then the package would parse timestamps like this instead:

select
    PARSE_TIMESTAMP('%Y-%m-%d %H:%M:%S %z', some_special_timestamp_column) AS parsed_timestamp

jcbmllgn avatar May 04 '23 14:05 jcbmllgn