sling-cli
sling-cli copied to clipboard
Source column not casted when number of rows selected in 0
We are running replication from PostgreSQL to BigQuery, e.g. something like that:
source: POSTGRESQL_X
target: BIGQUERY_X
streams:
dataset.table:
sql: >
SELECT *
FROM dataset.table
object: dataset.table
mode: full-refresh
source_options:
columns:
date: date
target_options:
table_keys:
partition: [date]
If SELECT returns empty rows, date column is not casted to DATE, and remains TIMESTAMP, which is a problem, as we partition by date in BQ.
Is there an option to skip sync if number of selected rows is 0 ? Or still cast column ?