sling-cli icon indicating copy to clipboard operation
sling-cli copied to clipboard

Source column not casted when number of rows selected in 0

Open ziga7631 opened this issue 9 months ago • 0 comments

We are running replication from PostgreSQL to BigQuery, e.g. something like that:

source: POSTGRESQL_X
target: BIGQUERY_X

streams:
  dataset.table:
    sql: >
      SELECT *
      FROM dataset.table
    object: dataset.table
    mode: full-refresh
    source_options:
      columns:
        date: date
    target_options:
      table_keys:
        partition: [date]

If SELECT returns empty rows, date column is not casted to DATE, and remains TIMESTAMP, which is a problem, as we partition by date in BQ.

Is there an option to skip sync if number of selected rows is 0 ? Or still cast column ?

ziga7631 avatar Mar 20 '25 12:03 ziga7631